Is there an easy way to make something like the top to the bottom picture where all columns are next to each other in an unnested way, same level?
Add a comment
|
1 Answer
You can update the source struct by adding fields from the inner struct. Something like this:
# get all fields of source struct except the inner struct e_struct
source_cols = [col(f"source.{c}") for c in df.select(col("source.*")).columns if c != "e_struct"]
# get all fields of the inner struct e_struct
e_struct_cols = [col(f"source.e_struct.{c}") for c in df.select(col("source.e_struct.*")).columns]
# combine them
new_struct_cols = source_cols + e_struct_cols
# update source column
df = df.withColumn("source", struct(*new_struct_cols))