Handling large sqlite database on Android

Question

I am working on dedicated Android device that is used for finger print matching. The can be database on this device that has 500K records.

I have modified the ROM on these devices to increase the heap size using directive -

dalvik.vm.heapgrowthlimit=742m dalvik.vm.heapsize=812m

I am trying to pre-load about 250MB database from SQlite into memory and it takes incredible amount of time. I have implemented this code in java using the Android SQLite interfaces. It takes about 10 min for 32,000 records to be loaded, is using Android/Java interface a lost cause and I should be doing all of this in native C code?

Are there are any improvements one can do to speed cursor loading using Android Java interface?

I keep seeing these messages as Android is loading the records:

-03 09:18:53.802 D/dalvikvm( 5234): GC_FOR_ALLOC freed 18K, 3% free 305194K/313816K, paused 187ms, total 187ms
10-03 09:18:53.982 D/dalvikvm( 5234): GC_FOR_ALLOC freed 0K, 3% free 305213K/313816K, paused 178ms, total 178ms
10-03 09:18:53.982 I/dalvikvm-heap( 5234): Grow heap (frag case) to 300.238MB for 4963-byte allocation
10-03 09:18:54.163 D/dalvikvm( 5234): GC_FOR_ALLOC freed 19K, 3% free 305204K/313824K, paused 176ms, total 176ms
10-03 09:18:54.333 D/dalvikvm( 5234): GC_FOR_ALLOC freed <1K, 3% free 305222K/313824K, paused 178ms, total 178ms
10-03 09:18:54.333 I/dalvikvm-heap( 5234): Grow heap (frag case) to 300.246MB for 4389-byte allocation

Is it really necessary to preload stuff from the database? One of the reasons to use a db is for quick access and, unless you are going to be doing some serious processsing each time stuff is loaded, it might be simpler to simply reload. — Peri Hartman
– Peri Hartman, Commented Oct 3, 2013 at 16:26
"is using Android/Java interface a lost cause and I should be doing all of this in native C code?" -- I would expect that to take a long time in native code as well. 250MB is huge, and flash speeds are slow. Not to mention that I am uncertain if SQLite is considered part of the NDK. "Are there are any improvements one can do to speed cursor loading using Android Java interface?" -- possibly, but we cannot see your code, since you did not paste any into your question. Use Traceview to determine where your performance problems lie. — CommonsWare
– CommonsWare, Commented Oct 3, 2013 at 16:26

Michael Dillon · Accepted Answer · 2013-10-03 17:25:46Z

1

You won't need to change most of this data, therefore you should use cdb or one of its alternatives instead. This will speed up access to the db, and you can still use SQLite to store new or changed records. Actually you might be able to do even better on load time if you use a raw memory dump format for your data but that may make search harder. If search time is an issue, consider using Lucene for storing the data in RAM.

SQL is just the wrong tool for loading this kind of data, even if you were running it on a 16 core server with 32G of RAM, I would still recommend cdb or Lucene.

answered Oct 3, 2013 at 17:25

Michael Dillon

32.5k7 gold badges76 silver badges107 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Handling large sqlite database on Android

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related