NiFi merge CSV files using MergeRecord

Question

i have a stream of JSON records that i convert it into CSV record successfully with this instruction. but now i want to merge this CSV records into one CSV file. below is that flow:

at step 5 i face with around 9K csv record, how do i merge it into one csv file using MergeRecord processor?

my csv header:

field1,field2,field3,field4,field5,field6,field7,field8,field9,field10,field11

some of this fields may be null and vary in records.

Is that a requirement to merge it to exactly one file? That won't be a good approach, IMO. If not, please go through nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/…. Minimum Number of Records and Maximum Number of Records are the ones that ultimately decide how the merge happens. — Sivaprasanna Sethuraman
– Sivaprasanna Sethuraman, Commented May 30, 2018 at 11:03
@sivaprasanna-sethuraman not exactly one file, one file that follows from merge strategy (like: bin packing algorithm) — meh
– meh, Commented May 30, 2018 at 12:21
Please try it out and tell if something doesn't work as expected. — Sivaprasanna Sethuraman
– Sivaprasanna Sethuraman, Commented May 30, 2018 at 12:24
@sivaprasanna-sethuraman clearly i try this processor; record reader: csvreader and writer: CSVRecordSetWriter; then with default properties and same schema that i used in json2csv convert i face with this error: "Cannot write Schema Name As Attribute because the Schema Name is not known merge csv" — meh
– meh, Commented May 30, 2018 at 12:25
Configure CsvRecordSetWriter controller service and Schema Write Strategy to Do not write schema — Sivaprasanna Sethuraman
– Sivaprasanna Sethuraman, Commented May 30, 2018 at 12:26

Abhishek Jha · Accepted Answer · 2019-05-14 12:35:52Z

0

after this use UpdateAttribute configure it so that it can save the file with a filename and after that use putFile to store it to a specific location

answered May 14, 2019 at 12:35

Abhishek Jha

1

Sign up to request clarification or add additional context in comments.

Comments

Renison Correya · Accepted Answer · 2022-11-07 16:17:41Z

0

I had a similar problem and solved it by using RouteonAttribute processor. Hope this helps someone.

Below is how I configure the processor using ${merge.count:equals(1)}

answered Nov 7, 2022 at 16:17

Renison Correya

706 bronze badges

Collectives™ on Stack Overflow

NiFi merge CSV files using MergeRecord

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related