Python Pandas: Increase Maximum Number of Rows

Question

I am processing a large text file (500k lines), formatted as below:

S1_A16
0.141,0.009340221649748676
0.141,4.192618196894668E-5
0.11,0.014122135626540204
S1_A17
0.188,2.3292323316081486E-6
0.469,0.007928706856794138
0.172,3.726771730573038E-5

I'm using the code below to return the correlation coefficients of each series, e.g. S!_A16:

import numpy as np
import pandas as pd
import csv
pd.options.display.max_rows = None
fileName = 'wordUnigramPauseTEST.data'

df = pd.read_csv(fileName, names=['pause', 'probability'])
mask = df['pause'].str.match('^S\d+_A\d+')
df['S/A'] = (df['pause']
              .where(mask, np.nan)
              .fillna(method='ffill'))
df = df.loc[~mask]

result = df.groupby(['S/A']).apply(lambda grp: grp['pause'].corr(grp['probability']))
print(result)

However, on some large files, this returns the error:

Traceback (most recent call last):
  File "/Users/adamg/PycharmProjects/Subj_AnswerCorrCoef/GetCorrCoef.py", line 15, in <module>
    print(result)
  File "/Users/adamg/anaconda/lib/python2.7/site-packages/pandas/core/base.py", line 35, in __str__
    return self.__bytes__()
  File "/Users/adamg/anaconda/lib/python2.7/site-packages/pandas/core/base.py", line 47, in __bytes__
    return self.__unicode__().encode(encoding, 'replace')
  File "/Users/adamg/anaconda/lib/python2.7/site-packages/pandas/core/series.py", line 857, in __unicode__
    result = self._tidy_repr(min(30, max_rows - 4))
TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

I understand that this is related to the print statement, but how do I fix it?

EDIT: This is related to the maximum number of rows. Does anyone know how to accommodate a greater number of rows?

A useful trick here (if you're using ipython) is to %debug and find out where the problem is. Tricky to say without a reproducible example. — Andy Hayden
– Andy Hayden, Commented Apr 2, 2014 at 22:15
My guess would be bad data, try checking the data for type... — dartdog
– dartdog, Commented Apr 6, 2014 at 16:34
Nothing to do with the data. As I said in the Edit, it has to do with the maximum number of rows. — Adam_G
– Adam_G, Commented Apr 6, 2014 at 20:18

unutbu · Accepted Answer · 2014-04-10 16:26:47Z

5

+50

The error message:

TypeError: unsupported operand type(s) for -: 'NoneType' and 'int'

is saying None minus an int is a TypeError. If you look at the next-to-last line in the traceback you see that the only subtraction going on there is

max_rows - 4

So max_rows must be None. If you dive into /Users/adamg/anaconda/lib/python2.7/site-packages/pandas/core/series.py, near line 857 and ask yourself how max_rows could end up being equal to None, you'll see that somehow

get_option("display.max_rows")

must be returning None.

This part of the code is calling _tidy_repr which is used to summarize the Series. None is the correct value to set when you want pandas to display all lines of the Series. So this part of the code should not have been reached when max_rows is None.

I've made a pull request to correct this.

edited Apr 10, 2014 at 16:26

answered Apr 8, 2014 at 0:08

unutbu

886k197 gold badges1.9k silver badges1.7k bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python Pandas: Increase Maximum Number of Rows

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related