Spark 1.4 Scala Dataframe binary to hex

Question

I have this field in my dataframe: myField: binary (nullable = false) (originally a Java byte[16]).

How do I select it and show it as a hex in scala from a spark shell? Please include necessary imports.

what have you tried? do you need information about how to convert binary to hex? or how to apply this function to your RDD? basically... mydf.select("myField").rdd.map(binToHex(_)) — Dan Ciborowski - MSFT
– Dan Ciborowski - MSFT, Commented Aug 11, 2016 at 14:35
do I need to convert it to an rdd first? Is there an equivalent dataframe function? — Marsellus Wallace
– Marsellus Wallace, Commented Aug 11, 2016 at 14:37
something like mydf.select("myField").map(binToHex($"myField")), but my DF psuedocode is a bit rougher then my rdd, might not even need the select. — Dan Ciborowski - MSFT
– Dan Ciborowski - MSFT, Commented Aug 11, 2016 at 14:38
What's the import? I'm getting: error: not found: value binToHex (NOTE: I'm new to everything here) — Marsellus Wallace
– Marsellus Wallace, Commented Aug 11, 2016 at 14:40
you have to code that function up yourself. That why I asked what you were looking for, I know spark, but not how to convert a bin to hex off the top of my head — Dan Ciborowski - MSFT
– Dan Ciborowski - MSFT, Commented Aug 11, 2016 at 14:42

user6022341 · Accepted Answer · 2016-08-11 19:09:54Z

2

Try

import org.apache.spark.sql.functions.{col, hex}

df.select(hex(col("myField")))

answered Aug 11, 2016 at 19:09

community wiki

user6022341

Sign up to request clarification or add additional context in comments.

Comments

Matrix · Accepted Answer · 2021-02-04 15:44:33Z

0

So I generalized @user6022341 solution to the following function

  def castBinaryToHexString(inputSrcDF: DataFrame):DataFrame={
    var srcDF = inputSrcDF

    // get all column names
    val allCols = srcDF.schema.fields.map(f => f.name.toLowerCase())

    // get all columns with data type = BinaryType
    val dateCols = srcDF.schema.fields.filter(f => f.dataType == BinaryType).map(f => f.name.toLowerCase())

    // cast all binary types to hex string
    srcDF = srcDF.select(allCols.map(x => if (dateCols.contains(x)) concat(lit("0x"),hex(col(x))).alias(x) else col(x) ): _*)

    return srcDF
  }

answered Feb 4, 2021 at 15:44

Matrix

1,8401 gold badge19 silver badges21 bronze badges

Collectives™ on Stack Overflow

Spark 1.4 Scala Dataframe binary to hex

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related