2

I have been trying to read data from a large (~270 MB) text file in Java and store it in an ArrayList of Mats and 3D point arrays in order to do Flann-Based Matching. Although these methods work, it takes too much time (roughly 20-30 minutes) to process and store the data. Is there a way to speed up adding the values to the ArrayList?

My text file in the following format: -1.78894 -0.260657 -5.01526 54 15 2899.69 817.542 14 0 0 1 ... (repeats for 124 more ints)

This is my code using BufferedReader, where mDescriptors is an ArrayList of Mat objects and mPoints3d is a 2D ArrayList of 3D points.

public void loadData() {

    // initialize mDescriptors and mPoints3d by iterating through it;
    for(int i = 0; i < 256; i++)
    {
        mDescriptors.add(new Mat());
        mPoints3d.add(new ArrayList<Point3>());
    }

    try {
        // open the database data file and read from it
        Log.i(TAG, "Opening table.txt");
        File fTable = new File(Environment.getExternalStorageDirectory().getPath() + "/GlassProject/table.txt");
        BufferedReader br = new BufferedReader(new FileReader(fTable));

        String str;
        // keep reading data until there is no more data to read
        while((str = br.readLine()) != null)
        {
            String[] sa = str.split(" ");
            Point3 xyz = new Point3(Float.valueOf(sa[0]), Float.valueOf(sa[1]), Float.valueOf(sa[2]));
            int imgidx = Integer.valueOf(sa[3]);
            //int featidx = Integer.valueOf(sa[4]);
            // Iterate through the rest of the line to get the descriptor data (128 values)
            for(int i = 7; i < sa.length; i++)
            {
                mDes.put(0,i - 7,Integer.valueOf(sa[i]));
            }
            mDescriptors.get(imgidx - 1).push_back(mDes); // Index issues happen here
            mPoints3d.get(imgidx - 1).add(xyz);
        }
        Log.v(TAG, "Loaded the Database data!");
    }
    catch(IOException ex) {
        Log.e(TAG, "Error: could not open table");
        ex.printStackTrace();
    }
}
5
  • Use Traceview and determine where your time is being spent. Note that you may not have enough heap space for what you are trying to read in. Commented Nov 2, 2015 at 20:24
  • After using Traceview and if the file IO is the bottleneck - consider using a compressed file as your source. Commented Nov 2, 2015 at 20:28
  • check here, here and here (probably one is a duplicate, too) Commented Nov 2, 2015 at 20:52
  • Sorry for the late reply, I found that most of the time was spent trying to add values to the arraylist, especially in that inner for loop. Is there a faster way to add elements to an arraylist? Commented Nov 6, 2015 at 18:41
  • What ArrayList? Do you mean mDes? Where is it declared? Is it created with an iniital capacity of 128? Have you considered using an actual array? Commented Aug 11, 2016 at 1:55

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.