Split string to array of characters in Spark

Question

How to split string column into array of characters?

Input:

from pyspark.sql import functions as F
df = spark.createDataFrame([('Vilnius',), ('Riga',), ('Tallinn',), ('New York',)], ['col_cities'])
df.show()
# +----------+
# |col_cities|
# +----------+
# |   Vilnius|
# |      Riga|
# |   Tallinn|
# |  New York|
# +----------+

Desired output:

# +----------+------------------------+
# |col_cities|split                   |
# +----------+------------------------+
# |Vilnius   |[V, i, l, n, i, u, s]   |
# |Riga      |[R, i, g, a]            |
# |Tallinn   |[T, a, l, l, i, n, n]   |
# |New York  |[N, e, w,  , Y, o, r, k]|
# +----------+------------------------+

Shubham Sharma · Accepted Answer · 2022-06-09 17:21:17Z

5

You can use split with regex pattern having negative lookahead:

df.withColumn('split', F.split('col_cities', '(?!$)'))

+----------+------------------------+
|col_cities|split                   |
+----------+------------------------+
|Vilnius   |[V, i, l, n, i, u, s]   |
|Riga      |[R, i, g, a]            |
|Tallinn   |[T, a, l, l, i, n, n]   |
|New York  |[N, e, w,  , Y, o, r, k]|
+----------+------------------------+

answered Jun 9, 2022 at 17:21

Shubham Sharma

71.8k6 gold badges26 silver badges58 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

ZygD · Accepted Answer · 2022-06-10 04:12:00Z

2

split can be used by providing empty string '' as separator. However, it will return empty string as the last array's element. So then slice is needed to remove the last array's element.

split = "split(col_cities, '')"
split = F.expr(f'slice({split}, 1, size({split})-1)')

df.withColumn('split', split).show(truncate=0)
# +----------+------------------------+
# |col_cities|split                   |
# +----------+------------------------+
# |Vilnius   |[V, i, l, n, i, u, s]   |
# |Riga      |[R, i, g, a]            |
# |Tallinn   |[T, a, l, l, i, n, n]   |
# |New York  |[N, e, w,  , Y, o, r, k]|
# +----------+------------------------+

edited Jun 10, 2022 at 4:12

answered Jun 9, 2022 at 14:18

ZygD

24.8k41 gold badges107 silver badges144 bronze badges

Collectives™ on Stack Overflow

Split string to array of characters in Spark

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related