fast and efficient way to read tab separated file using java

Question

What is the most efficient way (In terms of time) to read a text file into a list of array. File are of size 100 mb to 2 gb. The file contains data in following formatted :

From      TO          time     

a         b      13 decc 2009
b         c      13 decc 2009
c         d      13 decc 2009
f         h      13 decc 2009
f         g      13 decc 2009

Edit: Following is code for reading file

public List<InputDataBean> readInputData() throws Exception{
        List<InputDataBean> dataSet = new ArrayList<InputDataBean>();
        FileInputStream fstream = null;
        BufferedReader br = null;
        try{
            fstream = new FileInputStream(filePath);
            br = new BufferedReader(new InputStreamReader(fstream));
            String strLine;
            Set<String> users = new TreeSet<String>();
            while ((strLine = br.readLine()) != null)   {
                InputDataBean data = validateRecord(strLine);
                if(data==null)
                    continue;
                dataSet.add(data);
                users.add(data.getFromName());
                users.add(data.getToName());
            }
            UserKeys.setUsers(users);

        }catch (Exception e){
            throw e;
        }finally{
            try {
                if(null!=br)
                    br.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
        return dataSet;
    }

After reading file I want to store into array not to database.

If any other better alternative for reading file? Is it good idea to call script from java program and read data using script and store into java array.

P.S.: I really appreciate if anybody can edit or improve tags.

Firstly, how exactly are you reading your files ? There is no sample codes that could allow anyone to use as sample for suggestions. Secondly, what is your expected standard ? — thotheolh
– thotheolh, Commented Dec 13, 2011 at 6:01
There are some questions asked in stackoverflow regarding the parsing of tab delimited files in Java. I found one here: stackoverflow.com/questions/1635764/… — thotheolh
– thotheolh, Commented Dec 13, 2011 at 6:03
what are you doing with the data? if it goes to a database, you should use a tool that your database provides (most databases do). storing about 2 GB of data into the heap (as you read the file) may not be a great idea... generally, buffered readers are fine if you have to do this in java. — aishwarya
– aishwarya, Commented Dec 13, 2011 at 6:05
@thotheolh: Thanks for suggestion. sorry I want to read file using efficient way(In term of time) — Raje
– Raje, Commented Dec 13, 2011 at 6:14

Thomas Mueller · Accepted Answer · 2011-12-13 07:29:56Z

3

Possibly wrapping a BufferedInputStream around the FileInputStream will further improve performance a bit (because reads will be buffered in multiples of 4 KB). You could also play a bit with the buffer size.

If you know it's just ASCII, you could avoid using a Reader and possibly avoid creating a String for each line.

If you have the time, I would compare the performance of your solution with existing CSV reader tools, such as the CSV tool from the H2 database (disclosure: I wrote it).

answered Dec 13, 2011 at 7:29

Thomas Mueller

50.4k16 gold badges123 silver badges136 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

fast and efficient way to read tab separated file using java

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related