How can I combine multiple dataframes horizontally, based on 1 column, in Pyspark? [duplicate]

Question

I want to combine these 3 dataframes, based on their ID columns, and get the below output. I am after a short way that I can use it for combining many more number of dataframes later.

Input:

+---+---+---+
| ID|  a|  b|
+---+---+---+
|  A|  1|  1|
|  B|  2|  2|
+---+---+---+
+---+---+---+
| ID|  c|  d|
+---+---+---+
|  A|  3|333|
|  B|  4|444|
+---+---+---+
+---+---+---+
| ID|  e|  f|
+---+---+---+
|  A|555|  5|
|  B|666|  6|
+---+---+---+

Output:

+---+---+---+---+---+---+---+
| ID|  a|  b|  c|  d|  e|  f|
+---+---+---+---+---+---+---+
|  A|  1|  1|  3|333|555|  5|
|  B|  2|  2|  4|444|666|  6|
+---+---+---+---+---+---+---+

Answer: For anyone who might find it useful later!

# create list of dataframes
list_df = [df1, df2, df3]

# merge all at once
df_all = reduce(lambda x, y: x.join(y, on="ID"), list_df)

This question may have already been answered here (unfortunately it looks like it's not possible, or difficult): stackoverflow.com/questions/49763009/… — Stephen
– Stephen, Commented Oct 15, 2022 at 3:00
Did I miss something? Is that just df1 inner join df2 inner join df3 based on id column? — Jonathan
– Jonathan, Commented Oct 15, 2022 at 3:14
@JonathanLam it is what you said, but I am after a shorter way that I don't have to bring all one after another. So, that I can use it later for more number of dataframes. — Zrb
– Zrb, Commented Oct 15, 2022 at 6:13

过过招 · Accepted Answer · 2022-10-15 03:16:11Z

0

The 3 dataframes can be jioned according to the ID column.

df = df1.join(df2, 'ID').join(df3, 'ID')

answered Oct 15, 2022 at 3:16

过过招

4,3372 gold badges7 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How can I combine multiple dataframes horizontally, based on 1 column, in Pyspark? [duplicate]

1 Answer 1

Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Linked

Related