pyspark regex extract all

Question

I have a dataframe like below.

id  | js                                    |
0   | bla var test bla ..                   |
1   | bla function RAM blob                 |
2   | function CPU blob blob                |
3   | thanks                                |
4   | bla var AWS and function twitter blaa |

I am trying to extract the next word after function or var

My code is here.

pattern3 = "(func)\s+(\w+)|(var)\s+(\w+)"

df = df.withColumn("js_extracted2", f.regexp_extract(f.col("js"),pattern3,4))

as it is capture only one word, the final row returns only AWS and not Twitter.

So I would like to capture all matching.

My spark version is less than 3,

so I tried df.withColumn('output', f.expr("regexp_extract_all(js, '(func)\s+(\w+)|(var)\s+(\w+)', 4)")).show()

but it returns only empty for all rows.

my expected output is

id  | js                                    | output
0   | bla var test bla ..                   | [test]
1   | bla function RAM blob                 | [RAM]
2   | function CPU blob blob                | [CPU]
3   | thanks                                | 
4   | bla var AWS and function twitter blaa | [AWS, twitter]

can you try with my input, I tried but not giving expected output — Pyd
– Pyd, Commented Mar 7, 2022 at 10:16

过过招 · Accepted Answer · 2022-03-07 10:58:05Z

3

You need to use four \ to form a regular expression.

df = df.withColumn("js_extracted2", F.expr(f"regexp_extract_all(js, '(function|var)\\\\s+(\\\\w+)', 2)"))
df.show(truncate=False)

answered Mar 7, 2022 at 10:58

过过招

4,3372 gold badges7 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

AutomatedChaos Over a year ago

Alternatively, you can make your string a raw string and you'll only need two `\` characters. More info: docs.python.org/3/library/re.html

过过招 Over a year ago

@AutomatedChaos Yes, Thanks. Another way is to adjust the ``spark.sql.parser.escapedStringLiterals``` parameter to true.

Pyd Over a year ago

can you help in escaping this regex to work with regex_extract_all (?:[\w$]+?\.)(?:[\w$]+?\.?)+

过过招 Over a year ago

Can you give complete requirements, including input and output samples.

Pyd Over a year ago

hi @过过招 , actually this solution works fine on my local but in my server the spark version is 2.4.7. May I know how can I do it on lower version.. error:

pyspark.sql.utils.AnalysisException: "Undefined function: 'regexp_extract_all'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.; line 1 pos 0"

wwnde · Accepted Answer · 2022-03-07 12:21:05Z

2

use a positive look behind ascertion (?<=i)j. That is extract j only if i is immediately to its left

df.withColumn('name', regexp_extract('js', '((?<=function|var)(\s\w+))',1)).show()

---+--------------------+-----+
| id|                  js| name|
+---+--------------------+-----+
|  0| bla var test bla ..| test|
|  1| bla function RAM...|  RAM|
|  2|function CPU blob...|  CPU|
|  3|             thanks |     |
|  4|bla var AWS and f...|  AWS|
+---+--------------------+-----+

answered Mar 7, 2022 at 12:21

wwnde

26.7k6 gold badges22 silver badges38 bronze badges

1 Comment

Pyd Over a year ago

actually the approved solution works fine on my local but in my server the spark version is 2.4.7. May I know how can I do it on lower version.. error: pyspark.sql.utils.AnalysisException: "Undefined function: 'regexp_extract_all'. This function is neither a registered temporary function nor a permanent function registered in the database 'default'.; line 1 pos 0"

anky · Accepted Answer · 2022-03-07 12:58:59Z

The question looks more like how to do this without using regexp_extract_all to me.

Here is another way without regex, just incase one has a problem of using regexp_extract_all:

noofwordstoextract = 1
df.withColumn("ArrayOfWords",F.split("js"," "))\
.withColumn("test",F.expr(f"""filter(transform(ArrayOfWords,(x,e)-> 
CASE WHEN x in ('var','function') 
THEN array_join(slice(ArrayOfWords,e+2,{noofwordstoextract}),' ') ELSE NULL END)
,y-> y is not NULL)""")).drop("ArrayOfWords").show()


+---+-------------------------------------+--------------+
|id |js                                   |test          |
+---+-------------------------------------+--------------+
|0  |bla var test bla ..                  |[test]        |
|1  |bla function RAM blob                |[RAM]         |
|2  |function CPU blob blob               |[CPU]         |
|3  |thanks                               |[]            |
|4  |bla var AWS and function twitter blaa|[AWS, twitter]|
+---+-------------------------------------+--------------+

This solution splits the strings into an array then checks the current element is in 'var' or 'function' and then extracts the next n words (here 1) and then joins them to form the original string (try with noofwordstoextract=2) for more details.

Collectives™ on Stack Overflow

pyspark regex extract all

3 Answers 3

5 Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

5 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related