Grouping csv data using NiFi processors

Question

I have a set of 4 columned csv data where records have same value for the first column for 5 rows.Then again the value remains same for the first column for the next 5 rows and so on.

Sample data:

a,21,51,xxx
a,22,52,xxx
a,23,53,xxx
a,24,54,xxx
a,25,55,xxx
b,21,61,yyy
b,22,62,yyy
b,23,63,yyy
b,24,64,yyy
b,25,65,yyy
...........

But sometimes the records come in arbitrarily as:

a,21,51,xxx
a,22,52,xxx
a,23,53,xxx
b,21,61,yyy
b,22,62,yyy
a,24,54,xxx
a,25,55,xxx
b,23,63,yyy
b,24,64,yyy
b,25,65,yyy
...........

Is there any way of grouping such data based on its first column using NiFi processors?

Any answers would be helpful.

Thanks

you wants to group the rows if first column of the row is same.Is this your use case right? — Mahendran V M
– Mahendran V M, Commented Mar 17, 2017 at 10:38

Bryan Bende · Accepted Answer · 2017-03-17 13:57:40Z

3

You should be able to do this with the RouteText processor using the Grouping Regular Expression, which says:

"Specifies a Regular Expression to evaluate against each line to determine which Group the line should be placed in. The Regular Expression must have at least one Capturing Group that defines the line's Group. If multiple Capturing Groups exist in the Regular Expression, the Group from all Capturing Groups. Two lines will not be placed into the same FlowFile unless the they both have the same value for the Group (or neither line matches the Regular Expression). For example, to group together all lines in a CSV File by the first column, we can set this value to "(.?),.". Two lines that have the same Group but different Relationships will never be placed into the same FlowFile."

I think you can use that in conjunction with a Matching Strategy of Matches Regular Expression and just use .* for that expression so that every line matches.

Then for the grouping expression use the example above to group by the first column (.?),.

answered Mar 17, 2017 at 13:57

Bryan Bende

18.7k1 gold badge33 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

R.Sangeetha Over a year ago

I used the above solution but with few changes.For grouping expression expression I used:(.*?),.*

R.Sangeetha Over a year ago

Thanks a lot for your timely help.

Collectives™ on Stack Overflow

Grouping csv data using NiFi processors

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related