spring batch using spring boot: Read arguments from config or command line and use them in job

Question

I am pretty new to spring technology. I am trying to build an ETL like app using spring batch with spring boot.

Able to run the basic job (read->process->write). Now, I want to read the arguments (like date, file name, type, etc) from a config file (later) or command line (can work with it now) and use them in my job.

Entry point:

// Imports
@SpringBootApplication
@EnableBatchProcessing
public class EtlSpringBatchApplication {

    public static void main(String[] args) {
        SpringApplication.run(EtlSpringBatchApplication.class, args);
    }

}

My batch configuration

// BatchConfig.java
// Imports
    @Autowired
    public JobBuilderFactory jobBuilderFactory;

    @Autowired
    public StepBuilderFactory stepBuilderFactory;

    @Autowired
    public MyDao myDao;

    @Bean
    public Job job() {
        return jobBuilderFactory
                .get("job")
                .incrementer(new RunIdIncrementer())
                .listener(new Listener(myDao))
                .flow(step1())
                .end()
                .build();
    }

    @Bean
    public Step step1() {
        return stepBuilderFactory.get("step1").<myModel, myModel>chunk(1000)
                .reader(Reader.reader("my_file_20200520.txt"))
                .processor(new Processor())
                .writer(new Writer(myDao))
                .build();
    }

I have basic steps steps.

Reader.java has method to read flat file.

public static FlatFileItemReader<MyModel> reader(String path) {......}

Processor.java has process method defined. I added a @BeforeStep to fetch some details from DB required for processing.

public class Processor implements ItemProcessor<MyModel, MyModel> {

    private static final Logger log = LoggerFactory.getLogger(Processor.class);
    private Long id = null;

    @BeforeStep
    public void getId(StepExecution stepExecution) {
        this.id = stepExecution.getJobExecution().getExecutionContext().getLong("Id");
    }

    @Override
    public MyModel process(MyModel myModel) throws Exception {
    }
}

Writer.java is implementing ItemWriter and write code.

Listener.java extends JobExecutionListenerSupport and has overridden methods afterJob and beforeJob. Basically tried to use executioncontext here in beforeJob.

@Override
public void beforeJob(JobExecution jobExecution) {
    log.info("Getting the id..");
    this.id = myDao.getLatestId();
    log.info("id retrieved is: " + this.id);
    jobExecution.getExecutionContext().putLong("Id", this.id);
}

Now, what I am looking for is:

The reader should get the file name from job arguments. i.e. when run the job, I should be able to give some arguments, one of them is file path.
Later some methods (like get id, etc) require few more variables which can be passed as arguments to job i.e. run_date, type, etc.

In short I am looking for a way to,

Pass job arguments to my app (run_date, type, file path etc)
Use them in reader and other places (Listener, Writer)

Can someone provide me what addiitons I should do in my BatchConfig.java and other places, to read the job parameters (from command line or config file, whichever is easy)?

Mahmoud Ben Hassine · Accepted Answer · 2020-05-26 09:04:08Z

2

Both Spring Batch and Spring Boot reference documentation show how to pass parameters to a job:

Moreover, Spring Batch docs explain in details and with code examples how to use those parameters in batch components (like reader, writer, etc):

Late Binding of Job and Step Attributes

answered May 26, 2020 at 9:04

Mahmoud Ben Hassine

32.1k5 gold badges38 silver badges67 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Mihir Over a year ago

Thanks, it helped me in digging further and working for my case.

Vignesh T I · Accepted Answer · 2020-05-22 05:00:45Z

1

You can read the value of the of the job parameters set from the config file inside the reader or other classes within the spring batch execution context. Below is a snippet for reference,

application.yml file can have the below config,

batch.configs.filePath: c:\test

You can add the filePath read from the config to your job parameters when you start the job. Snippet of the class,

// Job and Job Launcher related autowires..

@Value("${batch.configs.filePath}")
private String filePath;

// inside a method block,
JobParameters jobParameters = new JobParametersBuilder().addLong("JobID", System.currentTimeMillis())
            .addString("filePath", filePath).toJobParameters();

try {
    jobLauncher.run(batchJob, jobParameters);
} catch (Exception e) {
    logger.error("Exception while running a batch job {}", e.getMessage());
}

One of the ways to access the Job Parameters is to implement StepExecutionListener to your reader Class to make use of its Overridden methods beforeStep and afterStep. Similar implementations can be performed to other classes as well,

public class Reader implements ItemReader<String>, StepExecutionListener {

private String filePath;

@Override
public void beforeStep(StepExecution stepExecution) {

    try {
        filePath = (String) stepExecution.getJobExecution().getExecutionContext()
                .get("filePath");
    } catch (Exception e) {
        logger.error("Exception while performing read {}", e);
    }
}


@Override
public String read() throws Exception {
    // filePath value read from the job execution can be used inside read use case impl

}

@Override
public ExitStatus afterStep(StepExecution stepExecution) {
    return ExitStatus.COMPLETED;
}

}

answered May 22, 2020 at 5:00

Vignesh T I

8923 gold badges9 silver badges24 bronze badges

5 Comments

Mihir Over a year ago

Thanks for the input, will be trying as above. However, can you please update on where to put the code for Job launcher and setting job parameters part (the snippet which you have shown)? my entry point of class is "EtlSpringBatchApplication" as shown in my example which I use to run the app. Which class will have hte job launcher and its code?

Vignesh T I Over a year ago

You can set the job parameters on the place where you actually run the batch job

Mihir Over a year ago

So, the BatchConfig.java file i.e. where I have Beans for job and steps? If I read the config value there, where should I set the job parameters as it is mentioned: "// inside a method block," -> which method is that? I am not too familiar with entire spring context, so looking for some guidance on structure for such code. If you can please show the project structure for such cases i.e. mainly reading, setting etc of job parameters (not readers, processors, or writers)? Just the point upto where I can read config and set job parameters, and how to run it?

Vignesh T I Over a year ago

The shared snippet has a controller block, on invocation of the API the created job configs which was created are autowired and job is being launched. To know more on how to run and configure, you can refer this link spring.io/guides/gs/batch-processing

Vignesh T I Over a year ago

Thats really great to hear!

Collectives™ on Stack Overflow

spring batch using spring boot: Read arguments from config or command line and use them in job

2 Answers 2

1 Comment

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related