7

To store big matrix on disk I use numpy.memmap.

Here is a sample code to test big matrix multiplication:

import numpy as np
import time

rows= 10000 # it can be large for example 1kk
cols= 1000

#create some data in memory
data = np.arange(rows*cols, dtype='float32') 
data.resize((rows,cols))

#create file on disk
fp0 = np.memmap('C:/data_0', dtype='float32', mode='w+', shape=(rows,cols))
fp1 = np.memmap('C:/data_1', dtype='float32', mode='w+', shape=(rows,cols))

fp0[:]=data[:]
fp1[:]=data[:]

#matrix transpose test
tr = np.memmap('C:/data_tr', dtype='float32', mode='w+', shape=(cols,rows))
tr= np.transpose(fp1)  #memory consumption?
print fp1.shape
print tr.shape

res = np.memmap('C:/data_res', dtype='float32', mode='w+', shape=(rows,rows))
t0 = time.time()
# redifinition ? res= np.dot(fp0,tr) #takes 342 seconds on my machine, if I multiplicate matrices in RAM it takes 345 seconds (I thinks it's a strange result)
res[:]= np.dot(fp0,tr) # assignment ?
print res.shape
print (time.time() - t0)

So my questions are :

  1. How to restrict memory consumtion of aplication which is using this procedure to some value for example to 100Mb(or 1Gb or something else).Also I don't understand how to estimate memory consumtion of procedure (I think memory is only allocated when "data" variable is created, but how much memory used when we use memmap files?)
  2. Maybe there is some optimal solution for multiplication of big matrices stored on disk? For example maybe data not optimally stored on disk or readed from disk, not properly chached, and also dot product use only one core.Maybe I should use something like PyTables?

Also I interested in algorithms solving linear system of equations (SVD and others) with restricted memory usage. Maybe this algorithms called out-of-core or iterative and I think there some analogy like hard drive<->ram, gpu ram<->cpu ram, cpu ram<->cpu cache.

Also here I found some info about matrix multiplication in PyTables.

Also I found this in R but I need it for Python or Matlab.

Improve this question
10
  • "How to restrict memory consumtion of aplication which is using this procedure to some value for example to 100Mb" You mean that if the application tries to use more memory it should fail? Using psutil.set_rlimit it's easy, but AFAIK it works only on linux. Commented Oct 14, 2013 at 11:39
  • No, I mean that application must work as normal but use less than declared memory(generally speaking it will be more slowly with less memory, but it usefull when we want to restrict application memory usage or if we haven't enough memory to fit whole matrix). And I work on Windows. Commented Oct 14, 2013 at 11:53
  • Your res line does not make sense (and res is the largest array...). Reread the np.dot docstring, you will find something useful... Commented Oct 15, 2013 at 9:03
  • what do you mean? output value must be "C-contiguous" ? and np.memmap is not suitable for this? or what? Commented Oct 15, 2013 at 10:10
  • 1
    "Maybe there is some optimal solution for multiplication of big matrices stored on disk?" Hadoop es.wikipedia.org/wiki/MapReduce michael-noll.com/tutorials/… Commented Oct 15, 2013 at 11:44

2 Answers 2

Reset to default
5

Dask.array provides a numpy interface to large on-disk arrays using blocked algorithms and task scheduling. It can easily do out-of-core matrix multiplies and other simple-ish numpy operations.

Blocked linear algebra is harder and you might want to check out some of the academic work on this topic. Dask does support QR and SVD factorizations on tall-and-skinny matrices.

Regardless for large arrays, you really want blocked algorithms, not naive traversals which will hit disk in unpleasant ways.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

2

Consider using NumExpr for your processing: https://github.com/pydata/numexpr

... internally, NumExpr employs its own vectorized virtual machine that is designed around a chunked-read strategy, in order to efficiently operate on optimally-sized blocks of data in memory. It can handily beat naïve NumPy operations if tuned properly.

NumExpr may cover #2 in your breakdown of the issue. If you address #1 by using a streamable binary format, you can then the chunked-read approach when loading your data files – like so:

    with open('path/to/your-data.bin', 'rb') as binary:
        while True:
            chunk = binary.read(4096) # or what have you
            if not chunk:
                break

If that is too low-level for you, I would recommend you look at the HDF5 library and format: http://www.h5py.org – it’s the best solution for the binary serialization of NumPy-based structures that I know of. The h5py module supports compression, chunked reading, dtypes, metadata… you name it.

Good luck!

Improve this answer

3 Comments

I'm not quite understand where numexpr can be used in matrix multiplication, can your expand answer and provide example of usage? Yes I know about hdf5 and pytables but numpy.memmap is more convenient because it can be used as usual numpy array.
numpy.memmap is an all-or-none operation, yes?… I was suggesting hdf5 as it supports chunking and streaming I/O: hdfgroup.org/HDF5/doc/Advanced/Chunking/index.html – while simply loading the same data with HDF5 may be slower than memory-mapped files, doing a chunked read to a chunk-based processing loop may be more efficient in the manner you’re looking for.
Regarding NumExpr, I don’t know if it offers a dot-product operator but I do know that it will give you faster view operations – if you were to redo your array ops using NumExpr, you might not need e.g. the memory-consuming transposition.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.