Function over multiple Spark DataFrame columns at once

Question

I need to calculate a md5 hash over multiple dataframe columns at once.

Function

def md5 = udf((s: String) => toHex(MessageDigest.getInstance("MD5").digest(s.getBytes("UTF-8"))))
def toHex(bytes: Array[Byte]): String = bytes.map("%02x".format(_)).mkString("")

Example with one column

var test_df = load_df.as('a).select($"a.attr1", md5($"a.attr2").as("hash_key"))

+-------------+--------------------+
|     attr1   |            hash_key|
+-------------+--------------------+
|9/1/2015 0:23|7a2f516dad8f13ae1...|
|9/1/2015 0:31|339c72b1870c3a6be...|
|9/1/2015 0:19|7065847af7abc6bce...|
|9/1/2015 1:32|38c7276958809893b...|

The generation with one column (a.attr2) works pretty good but i can't find any good way to insert/concatenate multiple columns into the md5() function.

eliasah · Accepted Answer · 2017-07-11 14:01:41Z

6

You should use concat_ws as followed:

md5(concat_ws(",",$"a.attr2",$"a.attr3",$"a.attr4"))

Here is an example :

Seq(("a","b","c")).toDF("x","y","z").withColumn("foo", md5(concat_ws(",",$"x",$"y",$"z"))).show(false)
// +---+---+---+--------------------------------+
// |x  |y  |z  |foo                             |
// +---+---+---+--------------------------------+
// |a  |b  |c  |a44c56c8177e32d3613988f4dba7962e|
// +---+---+---+--------------------------------+

edited Jul 11, 2017 at 14:01

answered Jul 11, 2017 at 13:46

eliasah

40.5k12 gold badges128 silver badges159 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Raphael Roth · Accepted Answer · 2017-07-11 14:09:24Z

1

Personally, I would do the concatenation inside the UDF, this gives you more flexibility:

e.g. passing array of Strings:

val md5 = udf((arrs:Seq[String]) => {
  val s = arrs.mkString(",")
  // do something with s
  s
 })    

df.withColumn("md5",md5(array($"x",$"y",$"z")))

Or even passing the entire row, which would also work if you have columns of mixed type:

val md5 = udf((r:Row) => {
  val s = r.mkString(",")
  // do something with s
  s
 })

df.withColumn("md5",md5(struct($"x",$"y",$"z")))

answered Jul 11, 2017 at 14:09

Raphael Roth

27.3k19 gold badges98 silver badges152 bronze badges

Comments

Abdul Mannan - Data Engineer · Accepted Answer · 2021-06-17 10:58:10Z

1

If you want to concate all columns using a custom delimiter, use this:

df.withColumn('row_hash', md5(concat_ws('||', *df.columns)))

Useful for calculating row hash.

answered Jun 17, 2021 at 10:58

Abdul Mannan - Data Engineer

1,11212 silver badges19 bronze badges

Collectives™ on Stack Overflow

Function over multiple Spark DataFrame columns at once

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related