read complex json file with spring batch

Question

i have a complex json file( with nested json arrays) structure like this:

{"persons":[ 
{"id":"1", "firstName": "X", "lastName": "X", "infos": [{"address":[{"city": "X", "country": "X"}]}]},
{"id":"2", "firstName": "Y", "lastName": "Y", "infos": [{"address":[{"city": "Y", "country": "Y"}]}]}
]}

I want to read each line ( one person) separately

So my spring batch configuration is like this

<bean id="reader" class="org.springframework.batch.item.file.FlatFileItemReader"
    scope="step">
    <property name="resource" value="#{jobParameters[file]}" />
    <property name="recordSeparatorPolicy" ref="recordPolicy" />
    <property name="lineMapper" ref="lineMapper" />
</bean>

<bean id="lineMapper"
    class="com.batchs.personJob.PersonLineMapper">
    <property name="delegate" ref="lineMapperType" />
</bean>

<bean id="lineMapperType"
    class="org.springframework.batch.item.file.mapping.JsonLineMapper" />

<bean id="recordPolicy"
    class="org.springframework.batch.item.file.separator.JsonRecordSeparatorPolicy" />

The mapper class looks like

    public class PersonLineMapper implements LineMapper<Person> {
    private JsonLineMapper delegate;

    public mapLine(String line, int lineNumber) throws Exception {
        Map<String, Object> personAsMap = delegate.mapLine(line, lineNumber);
        Person person = new Person();
        // map fields
        return person ;
    }

    public void setDelegate(JsonLineMapper delegate) {
        this.delegate = delegate;
    }
}

The problem is than the reader reads only one line (so one commit) because he reads the persons array in my json file like a whole line but i want read instead line per line ( one person at time)

How to do this ?

I have tried with a simple json file like this :

  { "id": "1",
      "firstName": "X",
      "lastName": "X"}
  { "id": "2",
      "firstName": "Y",
      "lastName": "Y"}

And it works good ... I read each person one by one

Thank you very much

Michael Minella · Accepted Answer · 2015-02-17 16:11:11Z

3

Unfortunately, you have two options:

Write your own record separator that ignores the wrapper persons element/line.
Edit the file in a step before reading it to remove that wrapper line using something like a sed command.

If you choose option 2, you can execute it via the SystemCommandTasklet in the step before the one you actually process it in.

answered Feb 17, 2015 at 16:11

Michael Minella

21.6k4 gold badges61 silver badges69 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

ulquiorra Over a year ago

Thank you very much , i will try out the first option and back with results( because i cannot modify the file ). Should i use the default RecordSeparatorPolicy interface or the JsonRecordSeparatorPolicy class ? Thanks

ulquiorra Over a year ago

Thank you very much , the first solution works . I manage to ignore the begin and the end of the file even if i did some makeshift job because the JsonRecordSeparatorPolicy unfortunately cannot handle json array by it's own

ams2705 Over a year ago

ulquiorra can you please share the solution

John Thompson · Accepted Answer · 2015-02-18 02:08:14Z

1

An alternative could be to run the file through a transform step to flatten the object structure, so you just have an array of People to deal with. Which should be fine, unless you're dealing with large files.

answered Feb 18, 2015 at 2:08

John Thompson

5243 silver badges7 bronze badges

1 Comment

ulquiorra Over a year ago

Thanks for your help . This solution works too ( i tried it ) but i will rather prefer use the record separator because my file is very large ( the step takes too long to proceed it )

Collectives™ on Stack Overflow

read complex json file with spring batch

2 Answers 2

3 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related