Get word frequency of pandas column containing lists of strings

Question

I have a pandas dataframe:

import pandas as pd
test = pd.DataFrame({'words':[['foo','bar none','scare','bar','foo'],
                              ['race','bar none','scare'],
                              ['ten','scare','crow bird']]})

I'm trying to get a word/phrase count of all the list elements in the dataframe colunn. My current solution is:

allwords = []

for index, row in test.iterrows():
    for word in row['words']:
        allwords.append(word)

from collections import Counter
pd.Series(Counter(allwords)).sort_values(ascending=False)

This works, but I was wondering if there was a faster solution. Note: I'm not using ' '.join() because I don't want the phrases to be split into individual words.

Shubham Sharma · Accepted Answer · 2021-01-22 11:36:35Z

5

Let's try .hstack with .value_counts:

pd.value_counts(np.hstack(test['words']))

scare        3
foo          2
bar none     2
ten          1
bar          1
crow bird    1
race         1
dtype: int64

answered Jan 22, 2021 at 11:36

Shubham Sharma

71.8k6 gold badges26 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Pygirl · Accepted Answer · 2021-01-22 13:35:13Z

3

Try using Counter:

import collections
words = test['words'].tolist()

collections.Counter([x for sublist in words for x in sublist])

Counter({'foo': 2,
         'bar none': 2,
         'scare': 3,
         'bar': 1,
         'race': 1,
         'ten': 1,
         'crow bird': 1})

answered Jan 22, 2021 at 13:35

Pygirl

13.4k6 gold badges36 silver badges48 bronze badges

Comments

jezrael · Accepted Answer · 2021-01-22 11:35:19Z

2

For improve performance dont use iterrows:

from collections import Counter
from  itertools import chain

a = pd.Series(Counter(chain.from_iterable(test['words']))).sort_values(ascending=False)
print (a)
scare        3
foo          2
bar none     2
bar          1
race         1
ten          1
crow bird    1
dtype: int64

Pandas only solution:

a = pd.Series([y for x in test['words'] for y in x]).value_counts()
print (a)
scare        3
bar none     2
foo          2
bar          1
race         1
crow bird    1
ten          1
dtype: int64

answered Jan 22, 2021 at 11:35

jezrael

868k103 gold badges1.4k silver badges1.3k bronze badges

Collectives™ on Stack Overflow

Get word frequency of pandas column containing lists of strings

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related