Getting "ambiguous reference" exception while reading nested json pyspark

Question

I have a below nested schema:

root
|-- fields: struct (nullable = true)
|    |-- configdata: struct (containsNull = true)
|    |    |-- field: string (nullable = true)
|    |    |-- type: string (nullable = true)
|    |    |-- value: string (nullable = true)
|    |-- configdata:struct (containsNull = true)
|    |    |-- field1: string (nullable = true)
|    |    |-- type1: string (nullable = true)
|    |    |-- value1: string (nullable = true)
|-- id: string (nullable = true)
|-- score: double (nullable = true)
|-- siteId: string (nullable = true)

I have to read both the configdata: property from this json. But when I am trying to do :

newDf= dataframe.select(sf.array(sf.expr("configdata"))

It is failing with the Exception:

Ambiguous reference to fields StructField(configdata)

As you can see the configData is of struct Type and I have to read both the config data from this json. Code developed in Pyspark using spark dataFrame API. Can someone please help?

Ged · Accepted Answer · 2021-03-03 18:10:30Z

2

There are 2 structs an no array situation evident. That is ambiguous as they are at the same level. Not possible. Give a new name to the second one: 'configdata1', at source.

edited Mar 3, 2021 at 18:10

answered Mar 3, 2021 at 17:47

Ged

18.5k8 gold badges53 silver badges108 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

sushant shekhar Over a year ago

Since I am new to pyspark, could you please help me editing the same names at the same level while parsing in pyspark only

Ged Over a year ago

U need to edit the source or ask them to supply a different name for the second struct at source.

Collectives™ on Stack Overflow

Getting "ambiguous reference" exception while reading nested json pyspark

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related