Spring-Batch: reading a file multi-format

Question

I'm trying to read a txt files using spring batch but my problem is each file has different data. for example each line of a file correspond to a Class, so for each line i need different FlatFileItemReader, Tokenizer, and FieldSetMapper.

My file looks like:

00|0|56||Class1|25|001|0
02|23|11||Class2|65|ENG|ENG|
02|32|25||Class3|45|0101|FRA|Es|TR

I've tried read file that contains same format of data and it works, but I don't know how to do it for file with different format.

FlatFileItemReader<Class1> fileReader=new FlatFileItemReader<Class1>();
fileReader.setResource(new ClassPathResource("/file.txt"));
DefaultLineMapper<Class1> lineMapper=new DefaultLineMapper<Class1>();
DelimitedLineTokenizer tokenizer=new DelimitedLineTokenizer();
tokenizer.setDelimiter("|");
lineMapper.setLineTokenizer(tokenizer);
fileReader.setLineMapper(lineMapper);

Any help would be appreciated . Thanks

Possible duplicate of How to read CSV file with different number of columns with Spring Batch — Mahmoud Ben Hassine
– Mahmoud Ben Hassine, Commented Apr 2, 2019 at 17:34
@MahmoudBenHassine thank you but each line represent an object of class — noor
– noor, Commented Apr 3, 2019 at 8:22
That's not an issue. The answer in the question I linked uses a PatternMatchingCompositeLineMapper which can be used here too with patterns like *Class1* and *Class2*, etc. But I edited the answer by @Luca Basso Ricci with an example because his answer is correct and he deserves the credit ! — Mahmoud Ben Hassine
– Mahmoud Ben Hassine, Commented Apr 4, 2019 at 9:29

Mahmoud Ben Hassine · Accepted Answer · 2019-04-04 09:22:45Z

1

Create a kinda of CompositeLineMapper where you store a different LineMapper implementation specific for every type of class you need to manage.
This CompositeLineMapper, for every line in your file, will look-ahead discriminator column and dispatch to right LineMapper implementation. I can't give you code because I'm not using SB atm,so I left to you the implementation.

EDIT: Adding an example of how to classify items

Here is a composite line mapper based on a Classifier:

import org.springframework.batch.item.file.LineMapper;
import org.springframework.classify.Classifier;

public class ClassifierCompositeLineMapper implements LineMapper<Object> {

    private Classifier<String, LineMapper<?>> classifier;

    public ClassifierCompositeLineMapper(Classifier<String, LineMapper<?>> classifier) {
        this.classifier = classifier;
    }

    @Override
    public Object mapLine(String line, int lineNumber) throws Exception {
        return classifier.classify(line).mapLine(line, lineNumber);
    }
}

And here is how to use it:

import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;

import org.springframework.batch.item.file.LineMapper;
import org.springframework.batch.item.file.mapping.PassThroughLineMapper;
import org.springframework.classify.Classifier;

public class ClassifierCompositeLineMapperTest {

    private ClassifierCompositeLineMapper compositeLineMapper;

    @Before
    public void setUp() {
        Classifier<String, LineMapper<?>> classifier = new Classifier<String, LineMapper<? extends Object>>() {
            @Override
            public LineMapper<?> classify(String classifiable) {
                if (classifiable.contains("Class1")) {
                    return new Class1LineMapper();
                }

                if (classifiable.contains("Class2")) {
                    return new Class2LineMapper();
                }
                return new PassThroughLineMapper(); // or any other default line mapper
            }
        };
        compositeLineMapper = new ClassifierCompositeLineMapper(classifier);
    }

    @Test
    public void mapLine() throws Exception {
        Object line1 = compositeLineMapper.mapLine("00|0|56||Class1|25|001|0", 1);
        Assert.assertTrue(line1 instanceof Class1);
        Object line2 = compositeLineMapper.mapLine("02|23|11||Class2|65|ENG|ENG|", 2);
        Assert.assertTrue(line2 instanceof Class2);
    }

    static class Class1 {}
    static class Class1LineMapper implements LineMapper<Class1> {

        @Override
        public Class1 mapLine(String line, int lineNumber) throws Exception {
            return new Class1(); // TODO mapping logic
        }
    }

    static class Class2 {}
    static class Class2LineMapper implements LineMapper<Class2> {

        @Override
        public Class2 mapLine(String line, int lineNumber) throws Exception {
            return new Class2(); // TODO mapping logic
        }
    }

}

edited Apr 4, 2019 at 9:22

Mahmoud Ben Hassine

32.1k5 gold badges38 silver badges67 bronze badges

answered Apr 3, 2019 at 14:35

Luca Basso Ricci

18.5k2 gold badges51 silver badges75 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

noor Over a year ago

thank you for your help, just I have one more question: when I use FlatFileItemReader, I call it with any class?

Mahmoud Ben Hassine Over a year ago

This answer is correct, and since Luca can't give an example atm, I did it. @LucaBassoRicci please don't hesitate to modify/amend the example if necessary.

Luca Basso Ricci Over a year ago

@noor: check ClassifierCompositeItemWriter

Mahmoud Ben Hassine Over a year ago

I second Luca's comment, to classify items at writing time, ClassifierCompositeItemWriter is the way to go. These two components (PatternMatchingCompositeLineMapper and ClassifierCompositeItemWriter) are usually used together in combination, here is a similar question/answer: stackoverflow.com/a/51766820/5019386. Hope this help.

noor Over a year ago

MahmoudBenHassine, LucaBassoRicci thank you very much it works with ClassifierCompositeItemWriter

|

Collectives™ on Stack Overflow

Spring-Batch: reading a file multi-format

1 Answer 1

8 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related