1

I just want to see if anybody can see an error in what I'm doing before I go and open an issue...

Minimal example - first create a multi-index dataframe:

miindex = pd.MultiIndex.from_product([["x","y"], ["10","20"]],names=['row-foo', 'row-bar'])    
micol = pd.MultiIndex.from_product([['a','b','c'], ["1","2"]],names=['col-foo', 'col-bar'])
df = pd.DataFrame(index=miindex, columns=micol).sortlevel().sortlevel(axis=1)
df = df.fillna(value=3.14) 
df

This gives us a nice test multi-index with column and row level names:

Python multi-index with valid column level names

Now if I make a sparse matrix out of that and show it, the column level names are gone:

ds = df.to_sparse()
ds

Python sparse multi-index with missing column level names

And if I convert the sparse version back to dense those level names are still gone:

tf = ds.to_dense()
tf

Python dense multi-index with missing column level names

Now I AM aware that displaying the sparse version calls to_dense() but the loss appears to be happening at the conversion to sparse. I'm exploring moving to sparse to reduce memory usage for a code base and my attempts to access the levels within the sparse dataframe generate "KeyError: 'Level not found'"

Anyone know how to preserve column level names in a pandas sparse dataframe?

(Tests shown on pandas 0.17.0, also observed on 0.16.2)

3
  • Looks like a bug, please file an issue on github. Commented Nov 13, 2015 at 22:03
  • Will do! (when back at my desk) Thanks! Commented Nov 13, 2015 at 22:07
  • @AndyHayden - bug reported, Jeff said sparse doesn't get much attention so I put together fix code myself and it's been merged to master. Thanks for the sanity check! Commented Nov 19, 2015 at 2:19

1 Answer 1

2

This was a bug in pandas.

Issue tracker here:https://github.com/pydata/pandas/issues/11600

Pull request sent and fix code merged to master

Looks like fix will be in 0.17.1

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.