create row, column, data pandas dataframe from sparse matrix

Question

How can I create a sparse matrix in the format of COO and have the pandas dataframe not unnest to a dense layout but keep the COO format for row,column,data?

import numpy as np
import pandas as pd
from scipy.sparse import csr_matrix
from scipy.sparse import coo_matrix

a = np.eye(7)
a_csr = csr_matrix(a)
a_coo = a_csr.tocoo()
print(a_coo)
  (0, 0)    1.0
  (1, 1)    1.0
  (2, 2)    1.0
  (3, 3)    1.0
  (4, 4)    1.0
  (5, 5)    1.0
  (6, 6)    1.0

I.e. how can I obtain a pandas dataframe from this that does not unnest this to

pd.DataFrame.sparse.from_spmatrix(a_coo)

but keeps the row,column,data format as also visualized in the print operation?

Basically I want to get the pandas dataframe in the Matrix Market format (MTX). — Georg Heiler
– Georg Heiler, Commented Aug 6, 2021 at 10:52
really ;) ok this could have been so easy. Certainly nicer than the mmwrite workaround. Do you want to write this as an answer? — Georg Heiler
– Georg Heiler, Commented Aug 6, 2021 at 14:56

hpaulj · Accepted Answer · 2021-08-06 15:47:09Z

2

The values you want to put in the dataframe are available as

a_coo.row, a_coo.col, a_coo.data

answered Aug 6, 2021 at 15:47

hpaulj

233k14 gold badges260 silver badges392 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Georg Heiler · Accepted Answer · 2021-08-06 11:05:49Z

0

one possible workaround could be to use mtx serialization and interpreting the data as a CSV.

from scipy import io
io.mmwrite('sparse_thing', a_csr)
!cat sparse_thing.mtx

sparse_mtx_mm_df = pd.read_csv('sparse_thing.mtx', sep=' ', skiprows=3, header=None)
sparse_mtx_mm_df.columns = ['row', 'column', 'data_value']
sparse_mtx_mm_df

Is there a better (native, non serialization-baased) solution?

re_sparsed = coo_matrix((sparse_mtx_mm_df['data_value'].values, (sparse_mtx_mm_df.numpy_row.values, sparse_mtx_mm_df.numpy_column.values)))
re_sparsed.todense()

would then give back the initial numpy array

answered Aug 6, 2021 at 11:05

Georg Heiler

17.9k44 gold badges176 silver badges320 bronze badges

Collectives™ on Stack Overflow

create row, column, data pandas dataframe from sparse matrix

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related