How to use .loc over a column with lists?

Question

this is my df:

    feature_name    combo                           p_val   *
0   VC9             [rest_closed, immediate_recall] 0.0053  **
1   VC9             [rest_music, immediate_recall]  0.0345  *
2   VC9             [rest_wonder, rest_closed]      0.0010  ***
3   VC9             [rest_wonder, rest_music]       0.0043  **
4   VC9             [rest_wonder, rest_open]        0.0075  **
5   Theta           [rest_closed, immediate_recall] 0.0098  **
6   Theta           [rest_wonder, rest_closed]      0.0038  **
7   Theta           [statements, rest_closed]       0.0187  *
8   Gamma           [rest_closed, clock]            0.0230  *
9   Gamma           [rest_closed, d1]               0.0111  *
10  Gamma           [rest_closed, immediate_recall] 0.0155  *
11  Gamma           [rest_closed, nb1]              0.0396  *
12  Gamma           [rest_wonder, rest_closed]      0.0065  **
13  Gamma           [statements, rest_closed]       0.0175  *

I an trying to reach the p_val through the feature_name & combo. meaning - I want to insert for examplt 'VC9' and [rest_closed, immediate_recall] and to get the matching p_val. everything I tried failed... this is what I have right now -

for feature in features:
    
    for comb in combinations:
        pval = df_sigs.loc[(df_sigs['feature_name'].isin([feature])) & (df_sigs['combo'].isin([comb])), df_sigs['p_val']]

And this is the error I get:

TypeError: unhashable type: 'list'

(I want to print the p_val over a plot so I don't need it for more than the one loop)

when I tried other things I also got this error many time:

ValueError: Lengths must match to compare

(for example when I used np.where)

I seriously tried anything I can think of - creating another column of strings that combines the two elemnts of the lists.. unlisting, turning the list to a tuple. I feel like I am missing something very basic.

oskros · Accepted Answer · 2022-12-05 10:10:49Z

1

Instead of using .loc, you can use the in-built __getitem__ method of pandas, like so.

Further, to filter on a column including a list, you need to make a lambda mask to get true value for each row in the column

for feature in features:

    for comb in combinations:
        tmp = df[(df['feature_name'] == feature) & (df['combo'].apply(lambda x: x == comb))]
        pval = tmp['p_val'].values[0]

edited Dec 5, 2022 at 10:10

answered Dec 5, 2022 at 9:50

oskros

3,3452 gold badges13 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

user20646185 Over a year ago

it still gives me this error: ValueError: Lengths must match to compare

oskros Over a year ago

fixed it, should work for you now

AltunE · Accepted Answer · 2022-12-06 10:25:31Z

1

feature_name = 'VC9'
combo = ['rest_closed', 'immediate_recall']

# Initialize the p_val variable to None
p_val = None

# Iterate over the rows of the DataFrame
for index, row in df.iterrows():
  # Check if the feature_name and combo match the given values
  if row['feature_name'] == feature_name and set(row['combo']) == set(combo):
    p_val = row['p_val']
    break

# Print the p_val
print(p_val)

edited Dec 6, 2022 at 10:25

answered Dec 5, 2022 at 9:51

AltunE

4106 silver badges12 bronze badges

2 Comments

user20646185 Over a year ago

not working :( [error message: ValueError: Lengths must match to compare]

AltunE Over a year ago

I'm sorry @user20646185. This error is happening because I m trying to compare a list of combo values with a single combo value. Maybe isin function could solve this issue. I have updated my code in a classic way (with a for loop, index). This one is working.

afzal7277 · Accepted Answer · 2022-12-05 10:03:07Z

0

df_sigs[(df_sigs['feature_name']=='VC9') & (df_sigs['combo']=="['rest_closed', 'immediate_recall']")].p_val.iloc[0]

answered Dec 5, 2022 at 10:03

afzal7277

11 bronze badge

1 Comment

Community Over a year ago

As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center.

Collectives™ on Stack Overflow

How to use .loc over a column with lists?

3 Answers 3

2 Comments

2 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related