Rewrite several matrix operations to avoid memory error?

Question

I have a very big matrix. buildmatrix is an N by N matrix. So sumHamiltonian is a N^2 by N^2 matrix. Then transform(N) is an N^2 by N(N-1)/2 matrix. (So the resultant shortHamiltonian(N) is an N(N-1)/2 by N(N-1)/2 matrix). The matrix entries are also complex numbers.

If N=200, I get a memory error. Is there a way to rewrite:

def sumHamiltonian(N):
    return 0.5*(np.kron(buildmatrix(N),np.identity(N))+np.kron(np.identity(N),buildmatrix(N)))

def shortHamiltonian(N):
    return np.matmul(np.transpose(transform(N)),np.matmul(sumHamiltonian(N),transform(N)))

to reduce memory?

I have seen some ways to reduce memory for matrix multiplication (Python/Numpy MemoryError) which is helpful but my memory error comes up at the kronecker product. Is there a way to rewrite this, or better yet, all of the matrix operations to avoid a memory error?

With a small N I make the arrays, and look for ways I break them up into chunks. Also the sum is a mean of 2 similar kron.. Those kron may be sparce due to np.idenity. I'd look for patterns specific to this case, rather hope for some generic 'memory saver'. N^4 gets big fast, especially if you are summing two. — hpaulj
– hpaulj, Commented Feb 24, 2020 at 5:39

kuzand · Accepted Answer · 2020-02-24 08:27:51Z

2

One suggestion is to use sparse matrices. Also, if possible, make your buildmatrix to return a matrix of dtype np.complex64 instead of np.complex128. See the example below:

from scipy import sparse


def buildmatrix(N):
    return (np.random.rand(N, N) + np.random.rand(N, N) * 1j).astype(np.complex64)

N = 100

m = buildmatrix(N)
I = np.identity(N)
sumHamiltonian = 0.5 * (np.kron(m, I) + np.kron(I, m))
print(f'{sumHamiltonian.nbytes * 1e-9} GB')
#1.6 GB

m_s = sparse.csr_matrix(m)
I_s = sparse.identity(N)
sumHamiltonian_s = 0.5 * (sparse.kron(m_s, I_s) + sparse.kron(I_s, m_s))
print(f'{sumHamiltonian_s.data.nbytes * 1e-6} MB')
#31.84 MB

#np.all(sumHamiltonian == sumHamiltonian_s)
#True

Similarly try to convert your shortHamiltonian to "sparse version".

edited Feb 24, 2020 at 8:27

answered Feb 24, 2020 at 8:22

kuzand

9,8864 gold badges48 silver badges50 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Janet Zhong Over a year ago

Sparse matrices has changed my life - thanks so much for the suggestion, this was exactly what I was after!

kuzand Over a year ago

Glad it helped! Please, consider accepting my answer.

ELinda · Accepted Answer · 2020-02-24 05:51:11Z

1

Depending on how much memory excess you have, you may be able to decrease the size of your matrices just by using something like .astype(np.float32) (e.g. if the current dtype is np.float64).

buildmatrix(N) is called twice in the same line (same with transform). If the result is deterministic (you just want to use the same result) then try assigning it to a variable, so that it only needs to be called once.

These tips might slightly reduce memory profile but might or might not be enough. I'm not sure if the computation can also be simplified by some identity.

answered Feb 24, 2020 at 5:51

ELinda

2,8211 gold badge13 silver badges9 bronze badges

Collectives™ on Stack Overflow

Rewrite several matrix operations to avoid memory error?

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related