1

I have following JSON, which I would like to convert into CSV.

Elements of JSON are going to be constant, if value is not present, then it will be null. But attribute will still be available.

I want to convert it into CSV with 9 columns.

I have flow like this => 

InvokeHTTP->EvaluateJSONPath->InferAvroSchema-> UpdateAttribute->ConvertRecord

last processor is failing.

a,b,c,d,e,sub-a,sub-b,sub-c,sub-d 

[
  {
    "a": "a-value",
    "b": "b-value",
    "c": "c-value",
    "d": "d-value",
    "e": 123,
    "sub_value": {
      "sub-a": "sub-a-value",
      "sub-b": null,
      "sub-c": "sub-c-value",
      "sub-d": "sub-d-value"
    }
  },
  {
    "a": "a2-value",
    "b": "b2-value",
    "c": "c2-value",
    "d": "d2-value",
    "e": 123,
    "sub_value": {
      "sub-a": "sub-2a-value",
      "sub-b": "sub-2a-value",
      "sub-c": "sub-2c-value",
      "sub-d": "sub-2d-value"
    }
  }
]

Update 2

What if, I do not want to use attributes in JsonReader? (i may have too many values, and it may become maintenance nightmare). is thr some other way?

Can I still solve my usecase by using avroschema while reading and writing?

input-json =>

{"store_number":"33152","store_name":"33152 WALMART JARDINES DEL COUNTRY 2374","store_display_name":"WALMART JARDINES DEL COUNTRY 2374","store_type_name":"Off Trade","street":"CALLE MORELOS 2019, JARDINES DEL COUNTRY, EL PERUL 2DA SECC.","city":"SALAMANCA","postal_code":"36764","latitude":20.665996,"longitude":-101.232345,"is_active":true,"manager_name":"MITZI ORTIZ","is_deleted":false,"manager_phone":null,"manager_email":null,"region_name":"A III","district_name":null,"branch_name":null,"retailer_name":"WALMART DE MEXICO","state_code":null,"store_additional_attributes":{"additional_attribute_1":"SSS MAINSTREAM","additional_attribute_2":"MAINSTREAM","additional_attribute_3":"PROM.SEM.53","additional_attribute_4":"SSS","additional_attribute_5":"R G","additional_attribute_6":"[email protected]","additional_attribute_11":"XEL HA MASTACHE","additional_attribute_12":"CALDERON VANESSA"}}

avro-schema-text (for JsonTreeReader)=>

{"type":"record","name":"jsn_to_csv","fields":[{"name":"store_number","type":"string","doc":"Type inferred from '\"1001\"'"},{"name":"store_name","type":"string","doc":"Type inferred from '\"1001 BODEGA AURRERA CHIMALHUACAN 3762\"'"},{"name":"store_display_name","type":"string","doc":"Type inferred from '\"BODEGA AURRERA CHIMALHUACAN 3762\"'"},{"name":"store_type_name","type":["string","null"],"doc":"Type inferred from '\"Off Trade\"'"},{"name":"street","type":["string","null"],"doc":"Type inferred from '\"AV CHIMALHUACAN 428 COL: BENITO JUAREZ\"'"},{"name":"city","type":["string","null"],"doc":"Type inferred from '\"NEZAHUALCOYOTL\"'"},{"name":"postal_code","type":["string","null"],"doc":"Type inferred from '\"57000\"'"},{"name":"latitude","type":["double","null","int"],"doc":"Type inferred from '19.403'"},{"name":"longitude","type":["double","null","int"],"doc":"Type inferred from '-99.0078'"},{"name":"is_active","type":"boolean","doc":"Type inferred from 'true'"},{"name":"manager_name","type":["string","null"],"doc":"Type inferred from '\"GIOVANI GARCIA\"'"},{"name":"is_deleted","type":"boolean","doc":"Type inferred from 'false'"},{"name":"manager_phone","type":"null","doc":"Type inferred from 'null'"},{"name":"manager_email","type":"null","doc":"Type inferred from 'null'"},{"name":"region_name","type":"string","doc":"Type inferred from '\"A IV\"'"},{"name":"district_name","type":"null","doc":"Type inferred from 'null'"},{"name":"branch_name","type":"null","doc":"Type inferred from 'null'"},{"name":"retailer_name","type":["string","null"],"doc":"Type inferred from '\"WALMART DE MEXICO\"'"},{"name":"state_code","type":"null","doc":"Type inferred from 'null'"},{"name":"store_additional_attributes","type":{"type":"record","name":"store_additional_attributes","fields":[{"name":"additional_attribute_1","type":["null","string"],"doc":"Type inferred from '\"SSS EMC\"'","default":null},{"name":"additional_attribute_2","type":["null","string"],"doc":"Type inferred from '\"EMC\"'","default":null},{"name":"additional_attribute_3","type":["null","string"],"doc":"Type inferred from '\"PROM.SEM.259\"'","default":null},{"name":"additional_attribute_4","type":["null","string"],"doc":"Type inferred from '\"SSS\"'","default":null},{"name":"additional_attribute_5","type":["null","string"],"doc":"Type inferred from '\"FRANCISCO HERNANDEZ\"'","default":null},{"name":"additional_attribute_6","type":["null","string"],"doc":"Type inferred from '\"[email protected]\"'","default":null},{"name":"additional_attribute_11","type":["null","string"],"doc":"Type inferred from '\"GIOVANI GARCIA\"'","default":null},{"name":"additional_attribute_12","type":["null","string"],"doc":"Type inferred from '\"SOLIS MARCIAL ALAN\"'","default":null}]}}]}

JsonTreeReader

enter image description here

CSVRecordSetWriter - Avro Schema

{"type":"record","name":"jsn_to_csv","fields":[{"name":"store_number","type":"string"},{"name":"store_name","type":"string"},{"name":"store_display_name","type":"string"},{"name":"additional_attribute_1","default":null,"type":[{"type":"null"},{"type":"string"}]}]}

CSVRecordSetWriter enter image description here

output=>

store_number,store_name,store_display_name,additional_attribute_1
33152,33152 WALMART JARDINES DEL COUNTRY 2374,WALMART JARDINES DEL COUNTRY 2374,

last column value is blank, it seems. how can I give reference of nested JSON attribute while writing it?

Template code 

<?xml version="1.0" encoding="UTF-8" standalone="yes"?><template encoding-version="1.2"><description></description><groupId>e67d1c1e-38c8-1085-8460-d34d1eab4f96</groupId><name>DNU_r_writer</name><snippet><connections><id>a554ad4c-5255-38a8-0000-000000000000</id><parentGroupId>50098679-9a85-3379-0000-000000000000</parentGroupId><backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold><backPressureObjectThreshold>10000</backPressureObjectThreshold><destination><groupId>50098679-9a85-3379-0000-000000000000</groupId><id>7ad8214e-4359-3339-0000-000000000000</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>50098679-9a85-3379-0000-000000000000</groupId><id>89b6d4b6-6c38-34f5-0000-000000000000</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>350019df-a89c-3065-0000-000000000000</id><parentGroupId>50098679-9a85-3379-0000-000000000000</parentGroupId><backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold><backPressureObjectThreshold>10000</backPressureObjectThreshold><destination><groupId>50098679-9a85-3379-0000-000000000000</groupId><id>ac868965-be33-3190-0000-000000000000</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>success</selectedRelationships><source><groupId>50098679-9a85-3379-0000-000000000000</groupId><id>7ad8214e-4359-3339-0000-000000000000</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><connections><id>51283208-4111-3683-0000-000000000000</id><parentGroupId>50098679-9a85-3379-0000-000000000000</parentGroupId><backPressureDataSizeThreshold>1 GB</backPressureDataSizeThreshold><backPressureObjectThreshold>10000</backPressureObjectThreshold><destination><groupId>50098679-9a85-3379-0000-000000000000</groupId><id>9060f008-2053-3dc0-0000-000000000000</id><type>PROCESSOR</type></destination><flowFileExpiration>0 sec</flowFileExpiration><labelIndex>1</labelIndex><name></name><selectedRelationships>failure</selectedRelationships><source><groupId>50098679-9a85-3379-0000-000000000000</groupId><id>7ad8214e-4359-3339-0000-000000000000</id><type>PROCESSOR</type></source><zIndex>0</zIndex></connections><controllerServices><id>2b600734-09f6-3b38-0000-000000000000</id><parentGroupId>50098679-9a85-3379-0000-000000000000</parentGroupId><bundle><artifact>nifi-record-serialization-services-nar</artifact><group>org.apache.nifi</group><version>1.6.0</version></bundle><comments></comments><descriptors><entry><key>Schema Write Strategy</key><value><name>Schema Write Strategy</name></value></entry><entry><key>schema-access-strategy</key><value><name>schema-access-strategy</name></value></entry><entry><key>schema-registry</key><value><identifiesControllerService>org.apache.nifi.schemaregistry.services.SchemaRegistry</identifiesControllerService><name>schema-registry</name></value></entry><entry><key>schema-name</key><value><name>schema-name</name></value></entry><entry><key>schema-version</key><value><name>schema-version</name></value></entry><entry><key>schema-branch</key><value><name>schema-branch</name></value></entry><entry><key>schema-text</key><value><name>schema-text</name></value></entry><entry><key>Date Format</key><value><name>Date Format</name></value></entry><entry><key>Time Format</key><value><name>Time Format</name></value></entry><entry><key>Timestamp Format</key><value><name>Timestamp Format</name></value></entry><entry><key>CSV Format</key><value><name>CSV Format</name></value></entry><entry><key>Value Separator</key><value><name>Value Separator</name></value></entry><entry><key>Include Header Line</key><value><name>Include Header Line</name></value></entry><entry><key>Quote Character</key><value><name>Quote Character</name></value></entry><entry><key>Escape Character</key><value><name>Escape Character</name></value></entry><entry><key>Comment Marker</key><value><name>Comment Marker</name></value></entry><entry><key>Null String</key><value><name>Null String</name></value></entry><entry><key>Trim Fields</key><value><name>Trim Fields</name></value></entry><entry><key>Quote Mode</key><value><name>Quote Mode</name></value></entry><entry><key>Record Separator</key><value><name>Record Separator</name></value></entry><entry><key>Include Trailing Delimiter</key><value><name>Include Trailing Delimiter</name></value></entry><entry><key>csvutils-character-set</key><value><name>csvutils-character-set</name></value></entry></descriptors><name>CSVRecordSetWriter</name><persistsState>false</persistsState><properties><entry><key>Schema Write Strategy</key><value>no-schema</value></entry><entry><key>schema-access-strategy</key><value>schema-text-property</value></entry><entry><key>schema-registry</key></entry><entry><key>schema-name</key><value>jsn_to_csv</value></entry><entry><key>schema-version</key><value>${avro.schema}</value></entry><entry><key>schema-branch</key></entry><entry><key>schema-text</key><value>{"type":"record","name":"jsn_to_csv","fields":[{"name":"store_number","type":"string","doc":"Type inferred from '\"33152\"'"},{"name":"store_name","type":"string","doc":"Type inferred from '\"33152 WALMART JARDINES DEL COUNTRY 2374\"'"},{"name":"store_display_name","type":"string","doc":"Type inferred from '\"WALMART JARDINES DEL COUNTRY 2374\"'"},{"name":"additional_attribute_1","default":null,"type":[{"type":"null"},{"type":"string"}]}]}</value></entry><entry><key>Date Format</key></entry><entry><key>Time Format</key></entry><entry><key>Timestamp Format</key></entry><entry><key>CSV Format</key></entry><entry><key>Value Separator</key></entry><entry><key>Include Header Line</key></entry><entry><key>Quote Character</key></entry><entry><key>Escape Character</key></entry><entry><key>Comment Marker</key></entry><entry><key>Null String</key></entry><entry><key>Trim Fields</key></entry><entry><key>Quote Mode</key></entry><entry><key>Record Separator</key></entry><entry><key>Include Trailing Delimiter</key></entry><entry><key>csvutils-character-set</key></entry></properties><state>DISABLED</state><type>org.apache.nifi.csv.CSVRecordSetWriter</type></controllerServices><controllerServices><id>50827b89-5d05-3339-0000-000000000000</id><parentGroupId>50098679-9a85-3379-0000-000000000000</parentGroupId><bundle><artifact>nifi-record-serialization-services-nar</artifact><group>org.apache.nifi</group><version>1.6.0</version></bundle><comments></comments><descriptors><entry><key>schema-access-strategy</key><value><name>schema-access-strategy</name></value></entry><entry><key>schema-registry</key><value><identifiesControllerService>org.apache.nifi.schemaregistry.services.SchemaRegistry</identifiesControllerService><name>schema-registry</name></value></entry><entry><key>schema-name</key><value><name>schema-name</name></value></entry><entry><key>schema-version</key><value><name>schema-version</name></value></entry><entry><key>schema-branch</key><value><name>schema-branch</name></value></entry><entry><key>schema-text</key><value><name>schema-text</name></value></entry><entry><key>Date Format</key><value><name>Date Format</name></value></entry><entry><key>Time Format</key><value><name>Time Format</name></value></entry><entry><key>Timestamp Format</key><value><name>Timestamp Format</name></value></entry></descriptors><name>JsonTreeReader</name><persistsState>false</persistsState><properties><entry><key>schema-access-strategy</key><value>schema-text-property</value></entry><entry><key>schema-registry</key></entry><entry><key>schema-name</key></entry><entry><key>schema-version</key></entry><entry><key>schema-branch</key></entry><entry><key>schema-text</key><value>{"type":"record","name":"jsn_to_csv","fields":[{"name":"store_number","type":"string","doc":"Type inferred from '\"1001\"'"},{"name":"store_name","type":"string","doc":"Type inferred from '\"1001 BODEGA AURRERA CHIMALHUACAN 3762\"'"},{"name":"store_display_name","type":"string","doc":"Type inferred from '\"BODEGA AURRERA CHIMALHUACAN 3762\"'"},{"name":"store_type_name","type":["string","null"],"doc":"Type inferred from '\"Off Trade\"'"},{"name":"street","type":["string","null"],"doc":"Type inferred from '\"AV CHIMALHUACAN 428 COL: BENITO JUAREZ\"'"},{"name":"city","type":["string","null"],"doc":"Type inferred from '\"NEZAHUALCOYOTL\"'"},{"name":"postal_code","type":["string","null"],"doc":"Type inferred from '\"57000\"'"},{"name":"latitude","type":["double","null","int"],"doc":"Type inferred from '19.403'"},{"name":"longitude","type":["double","null","int"],"doc":"Type inferred from '-99.0078'"},{"name":"is_active","type":"boolean","doc":"Type inferred from 'true'"},{"name":"manager_name","type":["string","null"],"doc":"Type inferred from '\"GIOVANI GARCIA\"'"},{"name":"is_deleted","type":"boolean","doc":"Type inferred from 'false'"},{"name":"manager_phone","type":"null","doc":"Type inferred from 'null'"},{"name":"manager_email","type":"null","doc":"Type inferred from 'null'"},{"name":"region_name","type":"string","doc":"Type inferred from '\"A IV\"'"},{"name":"district_name","type":"null","doc":"Type inferred from 'null'"},{"name":"branch_name","type":"null","doc":"Type inferred from 'null'"},{"name":"retailer_name","type":["string","null"],"doc":"Type inferred from '\"WALMART DE MEXICO\"'"},{"name":"state_code","type":"null","doc":"Type inferred from 'null'"},{"name":"store_additional_attributes","type":{"type":"record","name":"store_additional_attributes","fields":[{"name":"additional_attribute_1","type":["null","string"],"doc":"Type inferred from '\"SSS EMC\"'","default":null},{"name":"additional_attribute_2","type":["null","string"],"doc":"Type inferred from '\"EMC\"'","default":null},{"name":"additional_attribute_3","type":["null","string"],"doc":"Type inferred from '\"PROM.SEM.259\"'","default":null},{"name":"additional_attribute_4","type":["null","string"],"doc":"Type inferred from '\"SSS\"'","default":null},{"name":"additional_attribute_5","type":["null","string"],"doc":"Type inferred from '\"FRANCISCO HERNANDEZ\"'","default":null},{"name":"additional_attribute_6","type":["null","string"],"doc":"Type inferred from '\"[email protected]\"'","default":null},{"name":"additional_attribute_11","type":["null","string"],"doc":"Type inferred from '\"GIOVANI GARCIA\"'","default":null},{"name":"additional_attribute_12","type":["null","string"],"doc":"Type inferred from '\"SOLIS MARCIAL ALAN\"'","default":null}]}}]}</value></entry><entry><key>Date Format</key></entry><entry><key>Time Format</key></entry><entry><key>Timestamp Format</key></entry></properties><state>ENABLED</state><type>org.apache.nifi.json.JsonTreeReader</type></controllerServices><processors><id>89b6d4b6-6c38-34f5-0000-000000000000</id><parentGroupId>50098679-9a85-3379-0000-000000000000</parentGroupId><position><x>640.8075097656251</x><y>217.50194427490328</y></position><bundle><artifact>nifi-standard-nar</artifact><group>org.apache.nifi</group><version>1.6.0</version></bundle><config><bulletinLevel>WARN</bulletinLevel><comments></comments><concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount><descriptors><entry><key>File Size</key><value><name>File Size</name></value></entry><entry><key>Batch Size</key><value><name>Batch Size</name></value></entry><entry><key>Data Format</key><value><name>Data Format</name></value></entry><entry><key>Unique FlowFiles</key><value><name>Unique FlowFiles</name></value></entry><entry><key>generate-ff-custom-text</key><value><name>generate-ff-custom-text</name></value></entry><entry><key>character-set</key><value><name>character-set</name></value></entry></descriptors><executionNode>ALL</executionNode><lossTolerant>false</lossTolerant><penaltyDuration>30 sec</penaltyDuration><properties><entry><key>File Size</key><value>0B</value></entry><entry><key>Batch Size</key><value>1</value></entry><entry><key>Data Format</key><value>Text</value></entry><entry><key>Unique FlowFiles</key><value>false</value></entry><entry><key>generate-ff-custom-text</key><value>{"store_number":"33152","store_name":"33152 WALMART JARDINES DEL COUNTRY 2374","store_display_name":"WALMART JARDINES DEL COUNTRY 2374","store_type_name":"Off Trade","street":"CALLE MORELOS 2019, JARDINES DEL COUNTRY, EL PERUL 2DA SECC.","city":"SALAMANCA","postal_code":"36764","latitude":20.665996,"longitude":-101.232345,"is_active":true,"manager_name":"MITZI ORTIZ","is_deleted":false,"manager_phone":null,"manager_email":null,"region_name":"A III","district_name":null,"branch_name":null,"retailer_name":"WALMART DE MEXICO","state_code":null,"store_additional_attributes":{"additional_attribute_1":"SSS MAINSTREAM","additional_attribute_2":"MAINSTREAM","additional_attribute_3":"PROM.SEM.53","additional_attribute_4":"SSS","additional_attribute_5":"R G","additional_attribute_6":"[email protected]","additional_attribute_11":"XEL HA MASTACHE","additional_attribute_12":"CALDERON VANESSA"}}</value></entry><entry><key>character-set</key><value>UTF-8</value></entry></properties><runDurationMillis>0</runDurationMillis><schedulingPeriod>1d</schedulingPeriod><schedulingStrategy>TIMER_DRIVEN</schedulingStrategy><yieldDuration>1 sec</yieldDuration></config><name>GenerateFlowFile</name><relationships><autoTerminate>false</autoTerminate><name>success</name></relationships><state>STOPPED</state><style/><type>org.apache.nifi.processors.standard.GenerateFlowFile</type></processors><processors><id>9060f008-2053-3dc0-0000-000000000000</id><parentGroupId>50098679-9a85-3379-0000-000000000000</parentGroupId><position><x>0.0</x><y>0.0</y></position><bundle><artifact>nifi-standard-nar</artifact><group>org.apache.nifi</group><version>1.6.0</version></bundle><config><bulletinLevel>WARN</bulletinLevel><comments></comments><concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount><descriptors><entry><key>Log Level</key><value><name>Log Level</name></value></entry><entry><key>Log Payload</key><value><name>Log Payload</name></value></entry><entry><key>Attributes to Log</key><value><name>Attributes to Log</name></value></entry><entry><key>attributes-to-log-regex</key><value><name>attributes-to-log-regex</name></value></entry><entry><key>Attributes to Ignore</key><value><name>Attributes to Ignore</name></value></entry><entry><key>attributes-to-ignore-regex</key><value><name>attributes-to-ignore-regex</name></value></entry><entry><key>Log prefix</key><value><name>Log prefix</name></value></entry><entry><key>character-set</key><value><name>character-set</name></value></entry></descriptors><executionNode>ALL</executionNode><lossTolerant>false</lossTolerant><penaltyDuration>30 sec</penaltyDuration><properties><entry><key>Log Level</key><value>info</value></entry><entry><key>Log Payload</key><value>false</value></entry><entry><key>Attributes to Log</key></entry><entry><key>attributes-to-log-regex</key><value>.*</value></entry><entry><key>Attributes to Ignore</key></entry><entry><key>attributes-to-ignore-regex</key></entry><entry><key>Log prefix</key></entry><entry><key>character-set</key><value>UTF-8</value></entry></properties><runDurationMillis>0</runDurationMillis><schedulingPeriod>0 sec</schedulingPeriod><schedulingStrategy>TIMER_DRIVEN</schedulingStrategy><yieldDuration>1 sec</yieldDuration></config><name>LogAttribute</name><relationships><autoTerminate>false</autoTerminate><name>success</name></relationships><state>STOPPED</state><style/><type>org.apache.nifi.processors.standard.LogAttribute</type></processors><processors><id>ac868965-be33-3190-0000-000000000000</id><parentGroupId>50098679-9a85-3379-0000-000000000000</parentGroupId><position><x>542.0735742187499</x><y>5.220422363281159</y></position><bundle><artifact>nifi-standard-nar</artifact><group>org.apache.nifi</group><version>1.6.0</version></bundle><config><bulletinLevel>WARN</bulletinLevel><comments></comments><concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount><descriptors><entry><key>Log Level</key><value><name>Log Level</name></value></entry><entry><key>Log Payload</key><value><name>Log Payload</name></value></entry><entry><key>Attributes to Log</key><value><name>Attributes to Log</name></value></entry><entry><key>attributes-to-log-regex</key><value><name>attributes-to-log-regex</name></value></entry><entry><key>Attributes to Ignore</key><value><name>Attributes to Ignore</name></value></entry><entry><key>attributes-to-ignore-regex</key><value><name>attributes-to-ignore-regex</name></value></entry><entry><key>Log prefix</key><value><name>Log prefix</name></value></entry><entry><key>character-set</key><value><name>character-set</name></value></entry></descriptors><executionNode>ALL</executionNode><lossTolerant>false</lossTolerant><penaltyDuration>30 sec</penaltyDuration><properties><entry><key>Log Level</key><value>info</value></entry><entry><key>Log Payload</key><value>false</value></entry><entry><key>Attributes to Log</key></entry><entry><key>attributes-to-log-regex</key><value>.*</value></entry><entry><key>Attributes to Ignore</key></entry><entry><key>attributes-to-ignore-regex</key></entry><entry><key>Log prefix</key></entry><entry><key>character-set</key><value>UTF-8</value></entry></properties><runDurationMillis>0</runDurationMillis><schedulingPeriod>0 sec</schedulingPeriod><schedulingStrategy>TIMER_DRIVEN</schedulingStrategy><yieldDuration>1 sec</yieldDuration></config><name>LogAttribute</name><relationships><autoTerminate>false</autoTerminate><name>success</name></relationships><state>STOPPED</state><style/><type>org.apache.nifi.processors.standard.LogAttribute</type></processors><processors><id>7ad8214e-4359-3339-0000-000000000000</id><parentGroupId>50098679-9a85-3379-0000-000000000000</parentGroupId><position><x>208.0796655273441</x><y>260.6401416015624</y></position><bundle><artifact>nifi-standard-nar</artifact><group>org.apache.nifi</group><version>1.6.0</version></bundle><config><bulletinLevel>WARN</bulletinLevel><comments></comments><concurrentlySchedulableTaskCount>1</concurrentlySchedulableTaskCount><descriptors><entry><key>record-reader</key><value><identifiesControllerService>org.apache.nifi.serialization.RecordReaderFactory</identifiesControllerService><name>record-reader</name></value></entry><entry><key>record-writer</key><value><identifiesControllerService>org.apache.nifi.serialization.RecordSetWriterFactory</identifiesControllerService><name>record-writer</name></value></entry></descriptors><executionNode>ALL</executionNode><lossTolerant>false</lossTolerant><penaltyDuration>30 sec</penaltyDuration><properties><entry><key>record-reader</key><value>50827b89-5d05-3339-0000-000000000000</value></entry><entry><key>record-writer</key><value>2b600734-09f6-3b38-0000-000000000000</value></entry></properties><runDurationMillis>0</runDurationMillis><schedulingPeriod>0 sec</schedulingPeriod><schedulingStrategy>TIMER_DRIVEN</schedulingStrategy><yieldDuration>1 sec</yieldDuration></config><name>ConvertRecord</name><relationships><autoTerminate>false</autoTerminate><name>failure</name></relationships><relationships><autoTerminate>false</autoTerminate><name>success</name></relationships><state>STOPPED</state><style/><type>org.apache.nifi.processors.standard.ConvertRecord</type></processors></snippet><timestamp>10/08/2018 06:01:02 UTC</timestamp></template>
Improve this question

1 Answer 1

Reset to default
6

Use ConvertRecord processor with

  • JsonPathReader as Record Reader

  • CsvSetWriter as Record Writer

JsonPathReader Configs:

As you are having static elements in the json so add new properties matching with the json path for all keys of the json message enter image description here

AvroSchemaRegistry Configs:

This schema needs to match with the properties that we have added in JsonPathReader controller service. enter image description here

CsvSetWriter Configs:

enter image description here

Input:

[
  {
    "a": "a-value",
    "b": "b-value",
    "c": "c-value",
    "d": "d-value",
    "e": 123,
    "sub_value": {
      "sub-a": "sub-a-value",
      "sub-b": null,
      "sub-c": "sub-c-value",
      "sub-d": "sub-d-value"
    }
  },
  {
    "a": "a2-value",
    "b": "b2-value",
    "c": "c2-value",
    "d": "d2-value",
    "e": 123,
    "sub_value": {
      "sub-a": "sub-2a-value",
      "sub-b": "sub-2a-value",
      "sub-c": "sub-2c-value",
      "sub-d": "sub-2d-value"
    }
  }
]

Output:

a,b,c,d,e,sub-a,sub-b,sub-c,sub-d
a-value,b-value,c-value,d-value,123,sub-a-value,,sub-c-value,sub-d-value
a2-value,b2-value,c2-value,d2-value,123,sub-2a-value,sub-2a-value,sub-2c-value,sub-2d-value

For template that i tried will be found in this link.

Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

@RakeshPrasad, i don't think we can give reference a nested key using avro schema, This is only possible by using UpdateRecord (or) JsonPathReader controller services and adding new properties that refers the nested path(as i mentioned in the answer above). To flatten the json you can use FlattenRecord processor then use UpdateRecord processor to swap the values to refer your desired name. nifi.apache.org/docs/nifi-docs/components/org.apache.nifi/…
yes, to keep avro-schema dynamic, i did FlattenJson -> InferAvroSchema -> RecordConver -> SplitRecord -> Replace text (literal) -> merger content (defragmented, to keep header on top). One change I had to do with FlattenJson, I avoided using '.' as a seperator, as avro doesnt play well with anything other than ''. And in my case key already had '', so i used 'C' (as all my header was in smaller case). but then reverted it back to at the end.
@RakeshPrasad, Great, Awesome job you did :), Thanks for letting us know about the issues and resolutions..!!

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.