Spark: Merge columns of the same dataframe without creating additional dataframes

Question

I have the following data frame

+--------------------+-------------------+-------------+
|                uid2|               uid1|    timestamp|
+--------------------+-------------------+-------------+
|a                   |b                  |1589505008851|
|c                   |d                  |1589505012502|
|e                   |f                  |1589505016153|
+--------------------+-------------------+-------------+

and I want to create something like that

+--------------------+-------------------+
|                uids|          timestamp|
+--------------------+-------------------+
|a                   |1589505008851      |
|c                   |1589505012502      |
|e                   |1589505016153      |
|b                   |1589505008851      |
|d                   |1589505012502      |
|f                   |1589505016153      |
+--------------------+-------------------+

so I would like to merge the uid1 and uid2 columns into one column. The columns have the exact same length and they are of the same data type. Can I do this without creating an additional dataframe and "unioning" the two? Just by referencing the columns?

Raphael Roth · Accepted Answer · 2020-09-17 12:17:54Z

1

use the explode / array -approach :

df
.select(explode(array($"uid1",$"uid2")).as("uids"),$"timestamp")
.show()

answered Sep 17, 2020 at 12:17

Raphael Roth

27.3k19 gold badges98 silver badges152 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Niko Over a year ago

Worked like a charm! Thanks a lot! I will accept the answer in 2 minutes.. Apparently there is a limit to this...

Shrey Jakhmola · Accepted Answer · 2020-09-17 12:23:56Z

0

You can do that by:

Generating sample data

val data = Seq(("a", "b","1589505008851"), ("c", "d","1589505008852"), ("e", "f","1589505008854"))

val rdd = spark.sparkContext.parallelize(data)

var df = rdd.toDF("uid2","uid1","timestamp")

Code which will transform the data as per your requirement.

df=df.select($"uid2".as("uids"),$"timestamp").union(df.select($"uid1".as("uids"),$"timestamp"))

answered Sep 17, 2020 at 12:23

Shrey Jakhmola

5323 silver badges9 bronze badges

Collectives™ on Stack Overflow

Spark: Merge columns of the same dataframe without creating additional dataframes

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related