Spring batch IncorrectTokenCountException reader accept multi colomn in csv file

Question

I've never used Spring Batch before, I have a csv file which I need to read all columns with a specific model but I have many exceptions.

The csv file has 8 fields. if all the five fields are there, the file processing works well. but if by mistake, if 8 fields are not entered, like below:

c1;c2;C3;C4;C5;C6;C7;C8 // IT WORKS c1;C3;C7 // NOT WORKS : Program crashes

The error is: Caused by: org.springframework.batch.item.file.transform.IncorrectTokenCountException: Incorrect number of tokens found in record: expected 3 actual 8.

My code :

@Bean
public Step Step2(StepBuilderFactory stepBuilders) throws IOException {
    System.out.println("cecStep2");
    return stepBuilders.get("fileReject")
            .<CSCivique, String>chunk(100)
            .reader(reader())
            .processor(processor2FileReject())
            .writer(writer2())
            .build();
}

@Bean
public FlatFileItemReader<CSCivique> reader() throws IOException{

    try {


    return new FlatFileItemReaderBuilder<CSCivique>().name("personItemReader")
                                .resource(new ClassPathResource(confFile ()))
                                .delimited()
                                .delimiter(";")
                                .names(new String[] { "c1", "c2", "c3", "c4", "c5", "c6", "c7", "c8" })
                                .targetType(CSCivique.class)
                                .strict(false)
                                .build();

    }catch(Exception e) {
        System.out.println("----------- Exception reader() --------------");
        return null ;
    }


}

I want to save lines in file reject like : c1;C3;C7: please check the number of fields

Thank you.

saurabh kumar · Accepted Answer · 2021-04-13 11:10:35Z

understand that this answer is coming 2 years late, recently faced this issue in which I wanted to support partial rows to be accepted and processed.

In Spring batch the DelimitedLineTokenizer tokenizer (And most other standard tokenizers in spring batch) extends the AbstractLineTokenizer class which has a property strict which if set too true will fail if the number of tokens dont match. Conversely if the property is set too false then it will ignore extra/missing tokens.

https://docs.spring.io/spring-batch/docs/current/api/org/springframework/batch/item/file/transform/AbstractLineTokenizer.html

This is how I configure my flat file reader in XML

<bean class="org.springframework.batch.item.file.FlatFileItemReader" id="add-item-reader" scope="step">
<property name="resource" value="${file.path}"/>
<property name="lineMapper">
    <bean class="org.springframework.batch.item.file.mapping.DefaultLineMapper">
        <property name="fieldSetMapper" ref="fset-mapper"/>
        <property name="lineTokenizer">
            <bean class="org.springframework.batch.item.file.transform.DelimitedLineTokenizer">
                <property name="delimiter" value="|"/>
                <property name="names" value="tokens...."/>
                <property name="strict" value="false"/>
            </bean>
        </property>
    </bean>
</property>

Mahmoud Ben Hassine · Accepted Answer · 2019-02-25 08:47:21Z

2

That's the normal behaviour as documented here. You configured the reader to expect 8 columns with:

.names(new String[] { "c1", "c2", "c3", "c4", "c5", "c6", "c7", "c8" })

So any line having a different number of tokens will be rejected.

What you can do is use a fault tolerant step and skip those lines with:

@Bean
public Step Step2(StepBuilderFactory stepBuilders) throws IOException {
    System.out.println("cecStep2");
    return stepBuilders.get("fileReject")
            .<CSCivique, String>chunk(100)
            .reader(reader())
            .processor(processor2FileReject())
            .writer(writer2())
            .faultTolerant()
            .skip(FlatFileParseException.class)
            .skipLimit(10)
            .build();
}

With this configuration, those lines will be skipped and your job will not fail. You can afterwards get the skip count with StepExecution#getSkipCount.

Otherwise, you can create your own LineTokenizer and tokenize the line as you need.

answered Feb 25, 2019 at 8:47

Mahmoud Ben Hassine

32.1k5 gold badges38 silver badges67 bronze badges

6 Comments

Monoem Youneb Over a year ago

Thank you Mahmoud, can I save those skipped lines in a reject file csv? have you example of code?

Mahmoud Ben Hassine Over a year ago

Yes, you can do this using a SkipListener. Here is an example: stackoverflow.com/a/54648302/5019386. If this helps, please accept the answer. Hope it helps :-)

Monoem Youneb Over a year ago

Hello Mahmoud, How can I read csv file in ArryLis and save lines skipped in csv file? Thanks.

Monoem Youneb Over a year ago

hello, put each column in a box to check its contents (null or not) and write this line in a rejection file.

Mahmoud Ben Hassine Over a year ago

ok I see. You can use the PassThroughFieldSetMapper which will give you the FieldSet (similar to JDBC ResultSet). The field set allows you to get the array of values with getValues method. To write skipped lines, you need a custom SkipListener, you can find an example here: stackoverflow.com/a/51985616/5019386 (Just replace System.out with writing to a file).

|

Collectives™ on Stack Overflow

Spring batch IncorrectTokenCountException reader accept multi colomn in csv file

2 Answers 2

Comments

6 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

6 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related