spark join two dataframe without common column

Question

Need to join two dataframes in pyspark. One dataframe df1 is like:

city  user_count_city  meeting_session
NYC    100             5
LA     200             10 
....

Another dataframe df2 is like:

total_user_count  total_meeting_sessions
1000               100

Need to calculate user_percentage and meeting_session_percentage so I need a left join, something like

df1 left join df2

How could I join the two dataframes since they do not have common key?

Take a look of solution from this post Joining two dataframes without a common column But this is not same as my case.

Expected results

city user_count_city meeting_session total_user_count total_meeting_sessions
NYC    100             5                     1000     100
LA     200             10                    1000     100
....

i think if you add monotonicallyIncreasingId and CROSS JOIN on both data frames, would this serve your purpose? — Vikramsinh Shinde
– Vikramsinh Shinde, Commented Feb 27, 2021 at 3:29
even no condition needs to be specified - stackoverflow.com/questions/42158598/… — Vikramsinh Shinde
– Vikramsinh Shinde, Commented Feb 27, 2021 at 3:30
what are you trying to achieve if you don't have any common column. If first Df has 5 rows and second has 2 rows what would be the output that you would expect — Nikunj Kakadiya
– Nikunj Kakadiya, Commented Feb 27, 2021 at 4:02

blackbishop · Accepted Answer · 2021-02-27 06:25:58Z

5

You are looking for a cross join:

result = df1.crossJoin(df2)

answered Feb 27, 2021 at 6:25

blackbishop

32.8k11 gold badges61 silver badges86 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

spark join two dataframe without common column

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related