Load Multiple CSV files into database using Spring Batch

Question

I want to load multiple CSV files into mysql database at single table using Spring Batch. The path of the files are derived from the following method.

    public List<String> getFilePath() {
    String inputPath = "E:\\input";
    List<String> inputCSVPaths = new ArrayList<String>();
    Map<String, List<String>> inputInfo = new HashMap<String, List<String>>();

    File inputFolder = new File(inputPath);
    File[] inputFiles = inputFolder.listFiles();
    for (File file : inputFiles) {
        inputCSVPaths.add(file.getAbsolutePath());
    }
    inputInfo.put("Introduction", inputCSVPaths);
    List<String> inputFile = inputInfo.get("Introduction");
    System.out.println("Input File :"+inputFile);
    return inputFile;

}

There are total 3 CSV files. But it reads only onle file and inserts data of only that CSV file. Is there wrong in getting resources.

@Autowired
private FilePathDemo filePathDemo;

 @Bean
public MultiResourceItemReader<Introduction> multiResourceItemReader() throws IOException {

    MultiResourceItemReader<Introduction> multiReader = new MultiResourceItemReader<Introduction>();
    ResourcePatternResolver patternResolver = new PathMatchingResourcePatternResolver();

    Resource[] resources;
    String filePath = "file:";
    List<String> path = filePathDemo.getFilePath();
    for (String introPath : path) {
        System.out.println("File Path of the Introduction CSV :" + introPath);

        resources = patternResolver.getResources(filePath + introPath);
        multiReader.setResources(resources);
    }

        FlatFileItemReader<Introduction> flatReader = new FlatFileItemReader<Introduction>();
    multiReader.setDelegate(flatReader);
    flatReader.setLinesToSkip(1);
    flatReader.setLineMapper(new DefaultLineMapper<Introduction>() {
        {
            setLineTokenizer(new DelimitedLineTokenizer() {
                {
                    setNames(new String[] { "id", "name", "age", "phoneNo"});
                }
            });
            setFieldSetMapper(new BeanWrapperFieldSetMapper<Introduction>() {
                {
                    setTargetType(Introduction.class);
                }
            });
        }
    });

    flatReader.close();
    multiReader.close();

    return multiReader;
}

you are reassigning resource inside the for loop you should not be adding it every resource to the array? — rick
– rick, Commented Jul 17, 2018 at 12:21

Mahmoud Ben Hassine · Accepted Answer · 2018-07-17 17:02:33Z

2

There are two issues with your configuration:

You are reassigning the resources array with a single file in the for loop. Hence, the MultiResourceItemReader will be configured with only one file.
You are calling the close method on the MultiResourceItemReader and the delegate FlatFileItemReader but you should not. Spring Batch will call those methods when the step is complete.

You can find an example of how to configure the MultiResourceItemReader here: https://docs.spring.io/spring-batch/4.0.x/reference/html/readersAndWriters.html#multiFileInput

answered Jul 17, 2018 at 17:02

Mahmoud Ben Hassine

32.1k5 gold badges38 silver badges67 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

PAA · Accepted Answer · 2024-09-19 15:08:30Z

This example will help to understand how to use MultiResourceItemReader. Hope this will help.

import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.item.ItemWriter;
import org.springframework.batch.item.file.FlatFileItemReader;
import org.springframework.batch.item.file.MultiResourceItemReader;
import org.springframework.batch.item.file.builder.MultiResourceItemReaderBuilder;
import org.springframework.batch.item.file.mapping.DefaultLineMapper;
import org.springframework.batch.item.file.transform.DelimitedLineTokenizer;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.Resource;

import com.example.domain.Customer;
import com.example.mapper.CustomerFieldSetMapper;

@Configuration
public class JobConfig {
    @Autowired
    private JobBuilderFactory jobBuilderFactory;

    @Autowired
    private StepBuilderFactory stepBuilderFactory;

    @Value("classpath*:/data/customer*.csv")
    private Resource[] inputFiles;

    @Bean
    public MultiResourceItemReader<Customer> multiResourceItemReader() {
        return new MultiResourceItemReaderBuilder<Customer>()
                .name("multiResourceItemReader")
                .delegate(customerItemReader())
                .resources(inputFiles)
                .build();
    }

    @Bean
    public FlatFileItemReader<Customer> customerItemReader() {
        DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();
        tokenizer.setNames("id", "firstName", "lastName", "birthdate");

        DefaultLineMapper<Customer> customerLineMapper = new DefaultLineMapper<>();
        customerLineMapper.setLineTokenizer(tokenizer);
        customerLineMapper.setFieldSetMapper(new CustomerFieldSetMapper());
        customerLineMapper.afterPropertiesSet();

        FlatFileItemReader<Customer> reader = new FlatFileItemReader<>();
        reader.setLineMapper(customerLineMapper);

        return reader;
    }

    @Bean
    public ItemWriter<Customer> customerItemWriter(){
        return items -> {
            for (Customer customer : items) {
                System.out.println(customer.toString());
            }
        };
    }
    
    @Bean
    public Step step1() {
        return stepBuilderFactory.get("step1")
                .<Customer, Customer>chunk(10)
                .reader(multiResourceItemReader())
                .writer(customerItemWriter())
                .build();
    }
    
    @Bean
    public Job job() {
        return jobBuilderFactory.get("job")
                .start(step1())
                .build();
    }
}

Collectives™ on Stack Overflow

Load Multiple CSV files into database using Spring Batch

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related