How merge three DataFrame in Scala

Question

How merge 3 DataFrame in Spark-Scala? I completly don't have any Idea how I can make this. On stackOverFlow I can't found similar example.

I have 3 similar DataFrames. The same name of Column, and the same number of them. Difference is only a value on rows.

DataFrame1:

+----+------+----+---+
|type| Model|Name|ID |
+----+------+----+---+
|  1 |wdasd |xyzd|111|
|  1 |wd    |zdfd|112|
|  1 |bdp   |2gfs|113|
+----+------+----+---+

DataFrame2:

+----+------+----+---+
|type| Model|Name|ID |
+----+------+----+---+
|  2 |wdasd |xyzd|221|
|  2 |wd    |zdfd|222|
|  2 |bdp   |2gfs|223|
+----+------+----+---+

DataFrame3:

+----+------+----+---+
|type| Model|Name|ID |
+----+------+----+---+
|  3 |AAAA  |N_AM|331|
|  3 |BBBB  |NA_M|332|
|  3 |CCCC  |MA_N|333|
+----+------+----+---+

And I want to this type of DataFrame

MergeDataFrame:

+----+------+----+---+
|type| Model|Name|ID |
+----+------+----+---+
|  1 |wdasd |xyzd|111|
|  1 |wd    |zdfd|112|
|  1 |bdp   |2gfs|113|
|  2 |wdasd |xyzd|221|
|  2 |wd    |zdfd|222|
|  2 |bdp   |2gfs|223|
|  3 |AAAA  |N_AM|331|
|  3 |BBBB  |NA_M|332|
|  3 |CCCC  |MA_N|333|
+----+------+----+---+

Possible duplicate of How can I combine(concatenate) two data frames with the same column name in java — philantrovert
– philantrovert, Commented Mar 15, 2018 at 10:14

Steve Robinson · Accepted Answer · 2018-03-15 09:37:48Z

13

Spark provides a union and unionAll. Looks like they are deprecating the unionAll function so I would use the union function as below:

dataFrame1.union(dataFrame2).union(dataFrame3)

Note that in order to union data frames the data frames must have the exact same column names in the exact same order.

See the spark docs here

answered Mar 15, 2018 at 9:37

Steve Robinson

4525 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How merge three DataFrame in Scala

DataFrame1:

DataFrame2:

DataFrame3:

MergeDataFrame:

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

DataFrame1:

DataFrame2:

DataFrame3:

MergeDataFrame:

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related