How to remove substring in pyspark

Question

I want to takeout any value which is before (Impressions). Ex if i have value YouTube TrueView for Reach (Impressions), I will need YouTube TrueView for Reach.

Another example is YouTube Bumper (Impressions) --> YouTube Bumper

I am currently using :

validated_df=validated_df.withColumn("MediaNm", when(col("MediaNm").like("%Impressions%"),F.regexp_extract(F.col("MediaNm"), r".*?\(", 0)).otherwise(validated_df.MediaNm))

I am getting blank as a result of this.

media_name->column YouTube TrueView for Reach (Impressions)->value YouTube Bumper (Impressions)->value Expected value that needed to be extract from above values are YouTube TrueView for Reach YouTube Bumper — user19814628
– user19814628, Commented Sep 30, 2022 at 9:23

Ric S · Accepted Answer · 2022-09-30 09:33:35Z

1

If I understood correctly, you just want to remove the string ' (Impressions)': for this, you just need a regexp_replace

validated_df.withColumn('MediaNm', F.regexp_replace('MediaNm', ' \(Impressions\)', ''))

+--------------------------+
|MediaNm                   |
+--------------------------+
|YouTube TrueView for Reach|
|YouTube Bumper            |
+--------------------------+

answered Sep 30, 2022 at 9:33

Ric S

9,3184 gold badges30 silver badges57 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to remove substring in pyspark

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related