pyspark regexp_replace replacing multiple values in a column

Question

I have the url https://www.youcustomizeit.com/p/Equations-Kids-Backpack-Personalized/301793\r in dataset. I want to remove https:// at the start of the string and \r at the end of the string.

Creating dataframe to replicate the issue

c = spark.createDataFrame([('https://www.youcustomizeit.com/p/Equations-Kids-Backpack-Personalized/301793\r',)], ['str'])

I tried below regexp_replace with pipe function. But it is not working as expected.

c.select(F.regexp_replace('str', 'https:// | \\r', '')).first()

Actual output: www.youcuomizei.comEquaion-Kid-Backack-Peronalized301793

Expected output: www.youcustomizeit.com/p/Equations-Kids-Backpack-Personalized/301793

I added an extra space before and after pipe. Once I removed that it worked c.select(F.regexp_replace("str", "https://|[\\r]","")).first() — mohan chakradhar v
– mohan chakradhar v, Commented Oct 21, 2022 at 22:46

iambdot · Accepted Answer · 2022-10-21 23:00:58Z

the "backslash"r (\r) is not showing in your original spark.createDataFrame object because you have to escape it. so your spark.createDataFrame should be. please note the double backslashes

c = spark.createDataFrame([("https://www.youcustomizeit.com/p/Equations-Kids-Backpack-Personalized/301793\\r",)], ['str'])

which will give this output:

+------------------------------------------------------------------------------+
|str                                                                           |
+------------------------------------------------------------------------------+
|https://www.youcustomizeit.com/p/Equations-Kids-Backpack-Personalized/301793\r|
+------------------------------------------------------------------------------+

your regex https://|[\\r] will not remove the \r . the regex should be

c = (c
    .withColumn("str", F.regexp_replace("str", "https://|[\\\\]r", "")) 
)

which will give this output:

+--------------------------------------------------------------------+
|str                                                                 |
+--------------------------------------------------------------------+
|www.youcustomizeit.com/p/Equations-Kids-Backpack-Personalized/301793|
+--------------------------------------------------------------------+

Collectives™ on Stack Overflow

pyspark regexp_replace replacing multiple values in a column

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related