PySpark Add new object in nested field if not exist

Question

Schema

 root
     |-- userId: string (nullable = true)
     |-- languageknowList: array (nullable = true)
     |    |-- element: struct (containsNull = false)
     |    |    |-- code: string (nullable = false)
     |    |    |-- description: string (nullable = false)
     |    |    |-- name: string (nullable = false)

The df has userId and languageknownList. Every user should know English, so English language is not present in languageknowList I have to add.

English
code: 10
description: English Language
name: English

Any one please help me.

mck · Accepted Answer · 2021-01-06 18:49:05Z

2

You can create a new array of structs column and concat to the existing column:

import pyspark.sql.functions as F

english = F.struct(F.lit('10').alias('code'),
                   F.lit('English Language').alias('description'), 
                   F.lit('English').alias('name')
                  )

df2 = df.withColumn(
    'languageknowList',
    F.when(
        ~F.array_contains(F.col('languageknowList'), english),
        F.concat(
            F.col('languageknowList'),
            F.array(english)
        )
    ).otherwise(
        F.col('languageknowList')
    )
)

edited Jan 6, 2021 at 18:49

answered Jan 6, 2021 at 18:05

mck

42.7k13 gold badges44 silver badges62 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

SuFi Over a year ago

If it not exist the only need to concatenate

mck Over a year ago

@SuFi sorry, I missed that requirement. See the edited code?

Collectives™ on Stack Overflow

PySpark Add new object in nested field if not exist

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related