Evaluate string in column of dataframe with a variable

Question

I have a dataframe of the following format:

Name           Factor     Expression   Year 
Hydro          0.075            <10    2010  
Hydro          0.075            >10    2010  
Hydro          0.075            <10    2011  
Hydro          0.075            >10    2011
Hydro          0.075            <10    2012

And the following variable: i=3.

I would like to filter the dataframe where the Expression column evaluates as true, when the variable i is on the left hand side of the string in the column of expression.

For example, the first row would evaluate to true as 3<10.

The resulting dataframe that I would like is:

Name           Factor     Expression   Year 
Hydro          0.075            <10    2010  
Hydro          0.075            <10    2011  
Hydro          0.075            <10    2012

Thank you for any help.

user3483203 · Accepted Answer · 2018-12-06 20:38:48Z

2

Pandas has a safer version of eval that supports a limited number of operations. Luckily, > and < work, and you can use this along with string concatenation:

i = '3'
idx = pd.eval(i + df.Expression)
df.loc[idx]

    Name  Factor Expression  Year
0  Hydro   0.075        <10  2010
2  Hydro   0.075        <10  2011
4  Hydro   0.075        <10  2012

As @coldspeed noted, the above approach only works on DataFrames that are < 100 rows*, which isn't ideal. He also proposed the following solution:

df[[pd.eval(f"{i}{j}") for j in df['Expression']]]

_{*The above limitation is discussed more in depth in the following question: AttributeError: 'PandasExprVisitor' object has no attribute 'visit_Ellipsis', using pandas eval}

edited Dec 6, 2018 at 20:38

answered Dec 6, 2018 at 20:29

user3483203

51.3k10 gold badges72 silver badges104 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

cs95 Over a year ago

Careful, pd.eval will only work if your DataFrame is 100 rows or smaller.

user3483203 Over a year ago

Hmm, did not know that, really limits things quite a bit

cs95 Over a year ago

If you want to generalise to any size, you should use df[[pd.eval(f"{i}{j}") for j in df['Expression']]] instead.

user3483203 Over a year ago

@coldspeed do you know where that limitation is documented? I just get an attribute error when it reaches that size and I don't see it in the documentation

cs95 Over a year ago

Yes! It is documented here: stackoverflow.com/questions/48008191/…

Collectives™ on Stack Overflow

Evaluate string in column of dataframe with a variable

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related