python: pandas apply function: InvalidIndexError

Question

I am using apply on my data frame my_df like below:

my_df['column_C'] = my_df.apply(lambda x : 'hello' if x['column_B'] is None else x['column_B'] )

I want:

  if x['column_B'] = None -> return 'hello'
  if x['column_B'] != None -> return x['column_B']

Then I got the following errors:

<ipython-input-31-aa087c9a635e> in <lambda>(x)
----> 1 my_df['column_C'] = my_df.apply(lambda x : 'hello' if x['column_B'] is None else x['column_B'] )

/usr/local/lib/python3.4/dist-packages/pandas/core/series.py in __getitem__(self, key)
    599         key = com._apply_if_callable(key, self)
    600         try:
--> 601             result = self.index.get_value(self, key)
    602 
    603             if not is_scalar(result):

/usr/local/lib/python3.4/dist-packages/pandas/indexes/base.py in get_value(self, series, key)
   2187             # python 3
   2188             if is_scalar(key):  # pragma: no cover
-> 2189                 raise IndexError(key)
   2190             raise InvalidIndexError(key)
   2191 

IndexError: ('column_B', 'occurred at index column_A')

Does anyone know what I did wrong here?

why not: my_df['column_C'] = my_df['column_C'].fillna('hello')? — MaxU - stand with Ukraine
– MaxU - stand with Ukraine, Commented Nov 15, 2016 at 21:36
This is certainly a smarter approach thanks! Still curious about what was wrong in the apply function though ... — Edamame
– Edamame, Commented Nov 15, 2016 at 21:39

Julien Marrec · Accepted Answer · 2016-11-15 22:12:58Z

2

You need to apply specifying axis=1 to apply it to each row, not each column. See the documentation on DataFrame.apply:

axis : {0 or 'index', 1 or 'columns'}, default 0

* 0 or 'index': apply function to each column
* 1 or 'columns': apply function to each row

In your current call, it cannot find x['column_B'] when it's really using a pd.Series that corresponds to column_A.

So if you use the following it will work.

my_df['column_C'] = my_df.apply(lambda x : 'hello' 
                                if x['column_B'] is None
                                else x['column_B'], axis=1)

Note: as pointed in the comment above, DataFrame.fillna is more appropriate for this task.

edited Nov 15, 2016 at 22:12

answered Nov 15, 2016 at 21:43

Julien Marrec

11.9k5 gold badges51 silver badges67 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

python: pandas apply function: InvalidIndexError

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related