Pyspark self-join with error "Resolved attribute(s) missing"

Question

While doing a pyspark dataframe self-join I got a error message:

Py4JJavaError: An error occurred while calling o1595.join.
: org.apache.spark.sql.AnalysisException: Resolved attribute(s) un_val#5997 missing from day#290,item_listed#281,filename#286 in operator !Project [...]. Attribute(s) with the same name appear in the operation: un_val. Please check if the right attribute(s) are used.;;

It is a simple dataframe self-join like the one below, that works fine but after a couple of operations on the dataframe like adding columns or joining with other dataframes the error mentioned above is raised.

df.join(df,on='item_listed')

Using dataframe aliases like bellow wont work either and the same error message is raised:

df.alias('A').join(df.alias('B'), col('A.my_id') == col('B.my_id'))

Maviles · Accepted Answer · 2019-07-02 18:24:08Z

6

I've found a Java workaround here SPARK-14948 and for pyspark is like this:

#Add a "_r" suffix to column names array
newcols = [c + '_r' for c in df.columns]

#clone the dataframe with columns renamed
df2 = df.toDF(*newcols)

#self-join
df.join(df2,df.my_column == df2.my_column_r)

answered Jul 2, 2019 at 18:24

Maviles

3,4794 gold badges28 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Pyspark self-join with error "Resolved attribute(s) missing"

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related