0

Currently I need to implement the following steps with Spring batch:

  1. Read table data from data source A;
  2. Read table data from data source B based on the column values I get from data source A at step 1 as search criteria;
  3. Write what I get at step 2 to some other place

Technically I have no problem dealing with step 1 and step 3, but anyone could advise how to tackle step 2? I understand that after step 1 I can get a rowMapper class that maps each row of data to my domain object, in this case how to pass the column values (domain object attributes) as the parameters to step 2?

Improve this question
6
  • 1. Reader, 2. Processor, 3. writer... Basically the normal flow that Spring Batch follows. Create an ItemProcessor which looks up what you need from the database based on the input. Commented Dec 1, 2013 at 8:29
  • I am kinda new to spring batch, so I'd appreciate if you could be more specific. At step 2 I am going to do another select with column values I get at step 1 as where clause, you mean I can create an ItemProcessor where I can build up a sql and pass in the parameters from step 1? I used a JdbcPagingItemReader at step 1, my question is mainly on how to pass in the parameters at step 2....thanks. Commented Dec 1, 2013 at 23:44
  • Forget about steps, you can do everything in a single step (in the Spring Batch lingo that is). Your reader gets something, in your ItemProcessor simply use a JdbcTemplate to execute a query to get what you want from the database and pass it on to the ItemWriter. This is pretty much as clear as I can get, without implementing it for you. Commented Dec 2, 2013 at 6:29
  • sorry, dude, this didn't answer my question. as I mentioned my question is mainly on how to pass the parameters (from the reader to the processor in your way)? Commented Dec 2, 2013 at 15:00
  • Have you read the spring batch documentation? The result of the ItemReader is automatically passed to the ItemProcessor and that result is passed to the ItemWriter in a chunk oriented step. What you have is basically a single step in your sequence 1. is the ItemReader, 2. is the ItemProcessor and 3. is the ItemWriter. Commented Dec 2, 2013 at 15:24

2 Answers 2

Reset to default
2

Spring Batch Chunk Processing

As I tried to explain in the comments (and in the link to the documentation). Use a chunk oriented step. Your sequence corresponds to the following

  1. ItemReader
  2. ItemProcessor
  3. ItemWriter

For the reader you could use a JdbcCursorItemReader together with a RowMapper to convert the result into an object. In the ItemProcessor you use a JdbcTemplate with a query and use the incoming object to add the parameters to the query, together with another RowMapper this will convert the result into an object. This object is passed to the ItemWriter which stores the object you could use a JdbcBatchItemWriter for that.

Depending on your needs for step 2/3 you could try to create a custom writer which does the processing (reading/updating) in a single query (this could be faster then reading, constructing objects and writing again).

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

thanks, M.Deinum. After doing some investigation, I found it's the best implementation for me. Anyway SRT_KP's solution is also doable. Thank you both!
Hi, M.Deinum. If I put reader, processor and writer in one single step, how do I handle the transaction manager? I know I can only config one transaction manager in a tasklet, but I am reading from database A and write to database B, I suppose I need to create 2 separate transacton managers, right?
You can use a distributed transaction manager. Is it really a separate database? Also where do you store your batch tables?
it's like I create transactionManager_A for database A where I read, and create transactionManager_B for database B where I am to write. so when I config the job like below, which transaction manager I am going to use? obviously I cannot config 2 transaction managers in a tasklet. <job id="testJob" xmlns="springframework.org/schema/batch"> <step id="step1"> <tasklet transaction-manager="transactionManager_A"> <chunk reader="pagingItemReader" writer="jdbcItemWriter" commit-interval="20" /> </tasklet> </step> </job>
never mind, since the detailed business logic is still not determined yet for each step, I'll have time to read through the spring batch doc from scratch. Thank you, Deinum.
1

Why don't you create Staging table in Datasource B( I assume its different database). And tweek your query. Hence your step would be

  1. Step1: Read the data from DataSource A and write to staging table of Datasource B.
  2. Step2: Read the data from Datasouce B and write to file/some other place. (Tweek your select statement fetch in step2 to meet your conditions.
Improve this answer

1 Comment

yep, this can be the solution I am looking for.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.