Replace part of string in a Polars column with values from dictionary?

Question

Loving the Polars library for its fantastic speed and easy syntax!

Struggling with this question - is there an analogue in Polars for the Pandas code below? Would like to replace strings using a dictionary.

Tried using this expression, but it returns 'TypeError: 'dict' object is not callable'

pl.col("List").str.replace_all(lambda key: key,dict())

In Pandas I would use .replace()

import polars as pl

df = pl.DataFrame({'List': ['Systems', 'Software', 'Cleared']})

mapping = {'Systems':'Sys','Software':'Soft' ,'Cleared':'Clr'}

pl.from_pandas(df.to_pandas().replace(mapping, regex=True))

Output:

shape: (3, 1)
┌──────┐
│ List │
│ ---  │
│ str  │
╞══════╡
│ Sys  │
│ Soft │
│ Clr  │
└──────┘

I would think you're better off turning your lookup dict into another dataframe and then joining them — Dean MacGregor
– Dean MacGregor, Commented Dec 11, 2022 at 2:33

jqurious · Accepted Answer · 2024-07-06 14:56:30Z

4

There is a "stale" feature request for accepting a dictionary:

https://github.com/pola-rs/polars/issues/11418

One possible workaround is to stack multiple expressions in a loop:

expr = pl.col("List")

for old, new in dic.items():
    expr = expr.str.replace_all(old, new)
    
df.with_columns(result = expr)

shape: (3, 2)
┌──────────┬────────┐
│ List     ┆ result │
│ ---      ┆ ---    │
│ str      ┆ str    │
╞══════════╪════════╡
│ Systems  ┆ Sys    │
│ Software ┆ Soft   │
│ Cleared  ┆ Clr    │
└──────────┴────────┘

For non-regex cases, there is also .str.replace_many():

df.with_columns(
   pl.col("List").str.replace_many(
       ["Systems", "Software", "Cleared"],
       ["Sys", "Soft", "Clr"]
   )
   .alias("result")
)

edited Jul 6, 2024 at 14:56

answered Dec 9, 2022 at 2:32

jqurious

24.2k6 gold badges24 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Björn Over a year ago

Just wonder as this now nearly 1.5 years old is this still the best way to do a regex substituions? E.g. with SUBSTITUTIONS= {"_": " ","€": "eur", r"(?<![\w])eu(?![\w])": "eur" then for pat, val in SUBSTITUTIONS.items(): df = df.with_columns(pl.col('text').str.replace_all(pat,val)

jqurious Over a year ago

@Björn It depends. .str.replace_many now exists. (non-regex) Also, that regex wont work in this case as lookaround assertions are not supported by the underlying rust regex library.

Dean MacGregor · Accepted Answer · 2022-12-12 15:15:26Z

1

I think your best bet would be to turn your dic into a dataframe and join the two.

You need to convert your dic to the format which will make a nice DataFrame. You can do that as a list of dicts so that you have

dicdf=pl.DataFrame([{'List':x, 'newList':y} for x,y in dic.items()])

where List is what your column name is and we're arbitrary making newList our new column name that we'll get rid of later

You'll want to join that with your original df and then select all columns except the old List plus newList but renamed to List

df=df.join(
    dicdf, 
    on='List') \
.select([
    pl.exclude(['List','newList']), 
    pl.col('newList').alias('List')
 ])

answered Dec 12, 2022 at 15:15

Dean MacGregor

20.1k10 gold badges57 silver badges111 bronze badges

1 Comment

DBOak Over a year ago

Definitely an option, thanks

Collectives™ on Stack Overflow

Replace part of string in a Polars column with values from dictionary?

2 Answers 2

2 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related