strip numbers from pyspark dataframe column of type string

Question

I'm working on dataframe in pyspark. I've dataframe df and column col_1 which is array type and contains numbers as well.

Is there built in function to remove numbers from this string?

Dataframe schema:

>>> df.printSchema()
root
 |-- col_1: array (nullable = true)
 |    |-- element: string (containsNull = true)

Sample Values in Column:

>>>df.select("col_1").show(2,truncate=False)

+-------------------------------------------------------------------------------+
|col_1                                                                                                                                   
+-------------------------------------------------------------------------------+
|[use, bal, trans, ck, pay, billor, trans, cc, balances, got, grat, thnxs]                                                                  |
|[hello, like, farther, lower, apr, 11, 49, thank]|
+-------------------------------------------------------------------------------+

In this case, I'm looking for function which would strip number 11, 49 from second row. Thank you.

Pushkr · Accepted Answer · 2017-04-15 04:20:30Z

4

here is something you can try -

# Data preparation => 
data = [[['use', 'bal', 'trans', 'ck', 'pay', 'billor', 'trans', 'cc', 'balances', 'got', 'grat', 'thnxs']],
        [['hello', 'like', 'farther', 'lower', 'apr', '11', '49', 'thank']]]

df = sc.parallelize(data).toDF(["arr"])
df.printSchema()

:

root
 |-- arr: array (nullable = true)
 |    |-- element: string (containsNull = true)

:

from pyspark.sql.functions import explode,regexp_extract,col

df.select(explode(df.arr).alias('elements'))\
  .select(regexp_extract('elements','\d+',0)\
  .alias('Numbers'))\
  .filter(col('Numbers') != '').show()

Output :

+-------+
|Numbers|
+-------+
|     11|
|     49|
+-------+

answered Apr 15, 2017 at 4:20

Pushkr

3,63921 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

user2763088 Over a year ago

that worked. I was on vacation and couldn't test it. It worked fine.

user2763088 Over a year ago

Quick Sub Question: In my original ask was to strip numbers and keep charters only. I was finding syntax for keeping all characters so that my out put will be everything input string except numbers. Do you know syntax to define all charset in regexp_extract?

Pushkr Over a year ago

regex to find anything but digits would be \D+

user2763088 Over a year ago

ok, but I'm looking for syntax for filtering charters [ a to z].

user2763088 Over a year ago

For keep charters only : select(regexp_extract('elements','[a-zA-Z]+', 0))

Collectives™ on Stack Overflow

strip numbers from pyspark dataframe column of type string

1 Answer 1

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related