Pandas remove non-alphanumeric characters from string column

Question

with pandas and jupyter notebook I would like to delete everything that is not character, that is: hyphens, special characters etc etc

es:

firstname,birthday_date
joe-down§,02-12-1990
lucash brown_ :),06-09-1980
^antony,11-02-1987
mary|,14-12-2002

change with:

firstname,birthday_date
joe down,02-12-1990
lucash brown,06-09-1980
antony,11-02-1987
mary,14-12-2002

I'm trying with:

df['firstname'] = df['firstname'].str.replace(r'!', '')
df['firstname'] = df['firstname'].str.replace(r'^', '')
df['firstname'] = df['firstname'].str.replace(r'|', '')
df['firstname'] = df['firstname'].str.replace(r'§', '')
df['firstname'] = df['firstname'].str.replace(r':', '')
df['firstname'] = df['firstname'].str.replace(r')', '')

......
......
df

it seems to work, but on more populated columns I always miss some characters. Is there a way to completely eliminate all NON-text characters and keep only a single word or words in the same column? in the example I used firstname to make the idea better! but it would also serve for columns with whole words!

Thanks!

P.S also encoded text for emoticons

yudhiesh · Accepted Answer · 2021-09-30 13:54:01Z

3

You can use regex for this.

df['firstname'] = df['firstname'].str.replace('[^a-zA-Z0-9]', ' ', regex=True).str.strip()
df.firstname.tolist()
>>> ['joe down', 'lucash brown', 'antony', 'mary']

edited Sep 30, 2021 at 13:54

answered Sep 30, 2021 at 13:21

yudhiesh

6,8774 gold badges25 silver badges57 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

scofx Over a year ago

It seems to work, wouldn't it be better not to let space '[^a-zA-Z0-9]', ' ', with: '[^a-zA-Z0-9]', '', or it's the same thing ?

scofx Over a year ago

As unspoken ... the words remain united ... I thought it was the replacement space :)

balderman · Accepted Answer · 2021-09-30 13:18:01Z

1

Try the below. It works on the names you have used in post

first_names = ['joe-down§','lucash brown_','^antony','mary|']
clean_names = []
keep = {'-',' '}
for name in first_names:
    clean_names.append(''.join(c if c not in keep else ' ' for c in name if c.isalnum() or c in keep))
print(clean_names)

output

['joe down', 'lucash brown', 'antony', 'mary']

answered Sep 30, 2021 at 13:18

balderman

24k8 gold badges39 silver badges60 bronze badges

Collectives™ on Stack Overflow

Pandas remove non-alphanumeric characters from string column

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related