Python replacing partial matching text based on a list of elements in data frame

Question

I have built a dataframe that extracts data through a scraper. I extracted job positions, and currently, this column contains job positions as follows:

                                                Title Research Number  \
1                                                Dean             NaN   
2                                    Professor of Law             NaN   
3   Associate Dean for Information & Technology Se...             NaN   
4                                  Professor of Law\n             NaN   
5   Associate Dean for Faculty Development\nCharle...             NaN   
6   Associate Dean for Faculty Development\nCharle...             NaN   
7   Assistant Professor of Clinical Education & Di...             NaN   
8   Judge George Howard, Jr., Distinguished Profes...             NaN   
9                 Visiting Assistant Professor of Law             NaN   
10  Associate Dean for Academic Affairs\nArkansas ...             NaN   
11      Distinguished Professor in Constitutional Law             NaN   
12                         Assistant Professor of Law             NaN   
13  Instructor of Clinical Education; Supervising ...             NaN   
14                         Associate Professor of Law             NaN   
15                       Assistant Professor of Law\n             NaN   
16  Assistant Professor of Clinical Education; Tax...             NaN   
17         Assistant Professor of Law Librarianship;              NaN   
18  Byron M. Eiseman Distinguished Professor of Ta...             NaN   
19                                 Professor of Law\n             NaN   
20  Associate Professor of Law; Mediation Clinic D...             NaN   
21  Assistant Professor of Clinical Education; Fam...             NaN   
22   Assistant Professor of Clinical Education; Co...             NaN   
23                       Associate Professor of Law\n             NaN   
24  Professor of Law Librarianship; Electronic Res...             NaN   
25                                 Professor of Law\n             NaN   
26                                 Professor of Law\n             NaN   
27  Associate Dean for Experiential Learning & Cli...             NaN   
28                       Associate Professor of Law\n             NaN   
29  Assistant Professor of Clinical Education; Bus...             NaN   
30         Associate Professor of Law Librarianship;              NaN

I would like to replace these titles with the following titles:

titles=["Adjunct Professor","Professor Emeritus","Associate Professor","Assistant Professor","Professor"]

How can I look for partial text and replace it? I don't want to fully replace the text if it's not a 100% match. For example 'Visiting Assistant Professor of Law' should be replaced with 'Assistant Professor'

Thank you!

How do we know how much should be replaced? If you want to replace the whole value based on a partial match it would work. However, replacing only a few words based on a partial match is not easy as you do not know how much of the original text should be removed. — JANO
– JANO, Commented Mar 1, 2022 at 12:01
It's based on a partial match, let's say for the " Assistant Professor of..." the text should be replaced with "Assistant Professor", " Professor of..." would be replaced with "Professor". The rest of the text will be "deleted." If a partial match doesn't exist based on the list of titles, it would be replaced with a "NaN" — Shandron
– Shandron, Commented Mar 1, 2022 at 12:07

mozway · Accepted Answer · 2022-03-01 12:52:45Z

2

Use str.extract:

df['Title2'] = df['Title'].str.extract(f'({"|".join(titles)})')

output:

                  Title
1                   NaN
2             Professor
3                   NaN
...
29  Assistant Professor
30  Associate Professor

If you want to keep the original Title in case of no match, use:

df['Title'] = df['Title'].str.extract(f'({"|".join(titles)})', expand=False).fillna(df['Title'])

answered Mar 1, 2022 at 12:52

mozway

267k13 gold badges56 silver badges106 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python replacing partial matching text based on a list of elements in data frame

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related