How to use less memory to store data in dict python

Use a fast database that stores the key-value pairs to disk and allows intelligent retrieval and indexing, such as sqlite.

You should try using a database so you do not have to store all the data in memory.

A Berkeley Database is ideal for your use as it only stores key-value pairs. It is a "dict" in database form!

The code would look something like:

from bsddb3 import db
dbdict = DB()
dbdict.open("your database", None, db.DB_HASH, db.DB_CREATE)
dbdict[3]=2 #works just like a dict!

Here are bindings: Python "bindings" for Oracle Berkeley DB

Use pickle object to store data in dictionary. Refer this link to use pickle http://wiki.python.org/moin/UsingPickle

If the keys are integers, then, depending on the range of the keys, you can use an array http://docs.python.org/2/library/array.html instead of dictionary. Your keys become indices in the array, that's all. This will be more memory efficient than creating a dictionary.

If you don't have enough RAM to fit all your data into an array, then use something like sqlite or Berkeley DB to, effectively, have a dictionary on file. Of course, it will be much slower.

Since your indexes and data are integers, you can save your data in a file and access it as if it was an array, but only the pages you are working on will be in the RAM, the others will stay on the disk.

See http://docs.python.org/2/library/mmap.html

mmap is byte-based which means that an index in that will be like index*sizeof(int) on your architecture, and you will need to read sizeof(int) bytes instead of just one byte, and use the struct module (http://docs.python.org/2/library/struct.html) to convert that into a python integer.

This solution is a bit slower than using an array if all the data fits in the RAM, if your system starts paging out then this solution will be faster than using normal arrays.