TypeError: 'DataFrame' object is not callable - spark data frame

Question

i'm getting below error while executing the mentioned 'join' statement. i'm using pyspark setup. Any change required in the join statement or code.

TypeError: 'DataFrame' object is not callable

df11 = spark.read.option("header","true").option("delimiter", ",").csv("s3://mybucket/file1.csv")
df22 = spark.read.option("header","true").option("delimiter", ",").csv("s3://mybucket/file2.csv")
df11.createOrReplaceTempView("table1")
df22.createOrReplaceTempView("table2")
df1 = spark.sql( "select * from table1" )
df2 = spark.sql( "select * from table2" )

df_d = df1.join(df2, df1.NO == df2.NO, 'left').filter(F.isnull(df2.NO)).select(df1.NO,df1.NAME,df1.LAT,df1.LONG, F.lit('DELETE').alias('FLAG'))

Thanks

Ranga Vure · Accepted Answer · 2019-03-19 12:16:43Z

8

use the col names as string like this, it should work

df_d = df1.join(df2, df1['NO'] == df2['NO'], 'left').filter(F.isnull(df2['NO'])).select(df1['NO'],df1['NAME'],df1['LAT'],df1['LONG'], F.lit('DELETE').alias('FLAG'))

answered Mar 19, 2019 at 12:16

Ranga Vure

1,9323 gold badges16 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

cottontail · Accepted Answer · 2023-01-15 07:53:43Z

0

This will help to add new column from the list of existing column

for col_name in partition_key_list:
    print(col_name)
    #df_final_recs_i_u_n = df_final_recs_i_u_n.withColumn(f"{col_name}_partition_by", df_final_recs_i_u_n.date_tgt)
    df_final_recs_i_u_n=df_final_recs_i_u_n.withColumn(f"{col_name}_partition_by",df_final_recs_i_u_n[f"{col_name}_tgt"])

edited Jan 15, 2023 at 7:53

cottontail

25.6k25 gold badges184 silver badges176 bronze badges

answered Jan 15, 2023 at 7:50

Ranjan kumar Sejpada

211 bronze badge

Comments

Nikhil Baby · Accepted Answer · 2023-01-15 09:06:41Z

0

Once you have created the temporary view you can use Spark SQL to create the final dataframe. Please check the screenshot below:

Relevant SQL:

spark.sql("select table1.NO, table1.NAME, table1.LAT, table1.LONG, 'DELETE' as FLAG  from table1 left join table2 on table1.NO = table2.NO where table2.NO is null").show()

answered Jan 15, 2023 at 9:06

Nikhil Baby

8833 gold badges11 silver badges22 bronze badges

Collectives™ on Stack Overflow

TypeError: 'DataFrame' object is not callable - spark data frame

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related