Apply a custom function on columns in a pandas dataframe

Question

I want to do something equivalent to

Select x,y,z from data where f(x, Y);

And f is my customized function that looks into the values of specific columns in a row and returns True or False. I tried the following:

df = df.ix[_is_detection_in_window(df['Product'], df['CreatedDate'])== True]

But I get

TypeError: 'Series' objects are mutable, thus they cannot be hashed

I think it does not iterate over the rows. I also tried:

 i = 0
   for index, row in df.iterrows():
           if _is_detection_in_window(row['Product'], row['CreatedDate']):
                   print 'in range '
                   new_df.iloc[i] = row
                   i+= 1
   df = new_df

but I get :

IndexError: single positional indexer is out-of-bounds

cs95 · Accepted Answer · 2017-10-02 13:33:42Z

2

It seems like your function doesn't accept Series, but that can be changed using np.vectorize:

v = np.vectorize(_is_detection_in_window)
df = df.loc[v(df['Product'], df['CreatedDate'])]

Furthermore, you should refrain from using .ix which is now deprecated as of v20.

answered Oct 2, 2017 at 13:33

cs95

406k106 gold badges745 silver badges798 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Alex Over a year ago

yes, my function accepts two strings, so I would like to iterate over the rows and select the rows that match the function f that is defined on specific columns. I've written my other try in my post, but it does not work.

Alex Over a year ago

Perfect man!, awesome, thanks. Can you also tell me what is wrong with my second solution.

cs95 Over a year ago

@Alex Possibly an issue with your index. You might have wanted new_df.iloc[index] instead. Even so, if you want to filter out rows, what you're doing is incorrect.

PejoPhylo · Accepted Answer · 2017-10-02 13:33:38Z

0

Not sure how your function looks, but I assume it returns a list of bools equal to the number of rows in your df:

df = df.iloc[_is_detection_in_window(df['Product'], df['CreatedDate']), :]

answered Oct 2, 2017 at 13:33

PejoPhylo

4742 silver badges13 bronze badges

Collectives™ on Stack Overflow

Apply a custom function on columns in a pandas dataframe

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related