Why is scipy sparse matrix memory usage indifferent of the number of elements in the matrix?

Question

I have two scipy matrices 'a' and 'b' with boolean values in them. 'a' is way bigger than 'b': 765565 values against just 3 values.

In [211]: a
Out[211]: <388839x8455 sparse matrix of type '<class 'numpy.bool_'>'
           with 765565 stored elements in Compressed Sparse Row format>
In [212]: b
Out[212]: <5x3 sparse matrix of type '<class 'numpy.bool_'>'
           with 3 stored elements in Compressed Sparse Row format>

But when I check their sizes in terms of memory usage, I see that they are both just 56 bytes:

In [213]: from sys import getsizeof
          'Size of a: {}. Size of b: {}'.format(getsizeof(a), getsizeof(b))
Out[213]: 'Size of a: 56. Size of b: 56'

How come these matrices' sizes are the same, while matrix 'a' has to store over 200 thousand times more values than matrix 'b'?

See here getsizeof Return the size of an object in bytes. The object can be any type of object. All built-in objects will return correct results, but this does not have to hold true for third-party extensions as it is implementation specific. — Cory Kramer
– Cory Kramer, Commented Apr 28, 2017 at 13:24
@CoryKramer Thanks for the hint. a.data.nbytes works in this case. — Sergey Zakharov
– Sergey Zakharov, Commented Apr 28, 2017 at 13:45
@SergeyZakharov, don't forget to sum it up with a.indptr.nbytes + a.indices.nbytes — MaxU - stand with Ukraine
– MaxU - stand with Ukraine, Commented Apr 28, 2017 at 13:47

MaxU - stand with Ukraine · Accepted Answer · 2017-04-28 14:04:01Z

3

Here is a small demo:

from scipy import sparse

M = sparse.random(10**4, 10**3, .001, 'csr')

def sparse_memory_usage(mat):
    try:
        return mat.data.nbytes + mat.indptr.nbytes + mat.indices.nbytes
    except AttributeError:
        return -1

In [140]: sparse_memory_usage(np.random.rand(100, 100))
Out[140]: -1

In [141]: M = sparse.random(10**4, 10**3, .001, 'csr')

In [142]: sparse_memory_usage(M)
Out[142]: 160004

In [144]: M
Out[144]:
<10000x1000 sparse matrix of type '<class 'numpy.float64'>'
        with 10000 stored elements in Compressed Sparse Row format>

answered Apr 28, 2017 at 14:04

MaxU - stand with Ukraine

212k37 gold badges402 silver badges436 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Brecht Machiels Over a year ago

For COO sparse matrices, you need this: mat.data.nbytes + mat.row.nbytes + mat.col.nbytes.

Collectives™ on Stack Overflow

Why is scipy sparse matrix memory usage indifferent of the number of elements in the matrix?

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related