How to split list of dictionary in one column into two columns in pyspark dataframe?

Question

I want to split the filteredaddress column of the spark dataframe above into two new columns that are Flag and Address:

customer_id|pincode|filteredaddress|                                                              Flag| Address
1000045801 |121005 |[{'flag':'0', 'address':'House number 172, Parvatiya Colony Part-2 , N.I.T'}]
1000045801 |121005 |[{'flag':'1', 'address':'House number 172, Parvatiya Colony Part-2 , N.I.T'}]
1000045801 |121005 |[{'flag':'1', 'address':'House number 172, Parvatiya Colony Part-2 , N.I.T'}]

Can anyone please tell me how can I do it?

mck · Accepted Answer · 2021-02-18 16:45:19Z

1

You can get the values from filteredaddress map column using the keys:

df2 = df.selectExpr(
    'customer_id', 'pincode',
    "filteredaddress['flag'] as flag", "filteredaddress['address'] as address"
)

Other ways to access map values are:

import pyspark.sql.functions as F

df.select(
    'customer_id', 'pincode',
    F.col('filteredaddress')['flag'],
    F.col('filteredaddress')['address']
)

# or, more simply

df.select(
    'customer_id', 'pincode',
    'filteredaddress.flag',
    'filteredaddress.address'
)

edited Feb 18, 2021 at 16:45

answered Feb 18, 2021 at 16:05

mck

42.7k13 gold badges44 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

peeps Over a year ago

the above code is throwing error: cannot resolve 'from_json(filteredaddress)' due to data type mismatch: argument 1 requires string type, however, 'filteredaddress' is of map<string,string> type.

mck Over a year ago

@peeps the dataframe you showed in your question does not have a map type column. Could you do df.show() and copy the output to your question?

peeps Over a year ago

df.show() added the output to my question

Collectives™ on Stack Overflow

How to split list of dictionary in one column into two columns in pyspark dataframe?

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related