How to efficiently add sparse matrices in Python

Question

I want to know how to efficiently add sparse matrices in Python.

I have a program that breaks a big task into subtasks and distributes them across several CPUs. Each subtask yields a result (a scipy sparse matrix formatted as: lil_matrix).

The sparse matrix dimensions are: 100000x500000 , which is quite huge, so I really need the most efficient way to sum all the resulting sparse matrices into a single sparse matrix, using some C-compiled method or something.

A simple and efficient way to add sparse matrices is to convert them to sparse triplet form, concatenate the triplets, and then convert back to sparse column format. — David Heffernan
– David Heffernan, Commented Dec 30, 2010 at 19:45
Is the matrix addition for lil_matrices in NumPy not good enough? — Martin v. Löwis
– Martin v. Löwis, Commented Dec 30, 2010 at 20:54
no, it sounds pretty good enough. Actually this is my question. I just don't know how lil_matrix addition works - I couldn't find an example in the scipy site. If you could please give me an example that would solve my question — Alexandros
– Alexandros, Commented Dec 30, 2010 at 23:41

Hugh Bothwell · Accepted Answer · 2010-12-31 02:17:28Z

12

Have you tried timing the simplest method?

matrix_result = matrix_a + matrix_b

The documentation warns this may be slow for LIL matrices, suggesting the following may be faster:

matrix_result = (matrix_a.tocsr() + matrix_b.tocsr()).tolil()

answered Dec 31, 2010 at 2:17

Hugh Bothwell

57k9 gold badges91 silver badges103 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Vicky Liau Over a year ago

The second one is faster to me. matrix_result = numpy.array_a + sparse_matrix_b.tocsr(), compared to matrix_result = numpy.array_a + sparse_matrix_b

Wildhammer Over a year ago

A = lil_matrix((10000, 10000)) A[0, :100] = np.ones(100) A[1, :100] = np.ones(100) A[2, :100] = np.ones(100) A[100:300, 200] = np.ones((200,1)) A[100:300, 201] = np.ones((200,1)) A[100:300, 202] = np.ones((200,1)) A[100:300, 203] = np.ones((200,1)) B = lil_matrix((10000, 10000)) B[100, :50] = np.ones(50) B[101, :50] = np.ones(50) B[102, :50] = np.ones(50) B[100:200, 5000] = np.ones((100,1)) B[100:200, 5001] = np.ones((100,1)) B[100:200, 5002] = np.ones((100,1)) I tried this code and the first one is almost 4 times faster

Collectives™ on Stack Overflow

How to efficiently add sparse matrices in Python

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related