spring batch multithread same sort in reader as writer

Question

We are experimenting with a spring batch single thread to spring batch multithread.

The setup is rather easy :

Reader : Read some data with a specific sorting.
Processor : fetch some additional values.
Writer : Write it all to an csv file in order of read.

So we have changed the reader to a JdbcPagingItemReader and converted the sort from

order by firstname, lastname, id;

to

Map<String, Order> sortConfiguration = new HashMap<>();
sortConfiguration.put("firstname", Order.ASCENDING);
sortConfiguration.put("lastname", Order.ASCENDING);
sortConfiguration.put("id", Order.ASCENDING);

the commit-interval is set to 200.
The batch runs fine, but our csv is completly out of order.
I assumed that spring would write in the file after each commit (and hoped that he write page per page in order), but the disorder is greater then chunks 200 lines.
I got for example line 1, 3 and 5 should be together in a thread and line 2 and 4 in another thread.
Is there any option to preserve the order or is the only way to abandon multithread?

Mahmoud Ben Hassine · Accepted Answer · 2021-01-27 12:52:18Z

1

Multi-threading is incompatible with ordering. If you use a multi-threaded step, items will be read, processed and written in an undefined order. This is mentioned in the Multi-threaded Step section of the reference documentation:

The result of the above configuration is that the Step executes by reading, processing,
and writing each chunk of items (each commit interval) in a separate thread of execution.
Note that this means there is no fixed order for the items to be processed, and a chunk
might contain items that are non-consecutive compared to the single-threaded case.

answered Jan 27, 2021 at 12:52

Mahmoud Ben Hassine

32.1k5 gold badges38 silver badges67 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

spring batch multithread same sort in reader as writer

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related