Spark, how to read a file as byte array?

Question

Is there a way to read a file as byte array in spark?

As of now I am using the below code, but the content of the file changing in byte level. Its an encrypted file, so looking for ways to read the file without any change in byte level. I see lot of question on the same topic, but none provide a satisfactory answer. So posting this question as it could help other also. Thanks

val rawFileRDDEncrypted = spark.sparkContext.textFile("path")

No it doesn't resolve my question. I am trying to read only one ecrypted file as byte array — Minisha
– Minisha, Commented Mar 3, 2021 at 8:38
sc.binaryFile is what refered in the link you shared above and it didnt work — Minisha
– Minisha, Commented Mar 3, 2021 at 8:44
why it didn't work? The docs says it will read as a byte array. — mck
– mck, Commented Mar 3, 2021 at 8:45
spark.sparkContext.binaryFiles("path") this is what I tried and I couldnt get the bytes out of it. If you have snippets can you share ... thanks — Minisha
– Minisha, Commented Mar 3, 2021 at 8:52

Shaido · Accepted Answer · 2021-03-04 03:51:30Z

3

Made it work with this

val binaryFileList = spark.sparkContext.binaryFiles("file").collect()
val byteArray: Array[Array[Byte]] = binaryFileList.map(tuple=> {
  val pds = tuple._2
  val dis = pds.open()
  val len = dis.available();
  val buf = Array.ofDim[Byte](len)
  pds.open().readFully(buf)
  buf
})

edited Mar 4, 2021 at 3:51

Shaido

28.6k26 gold badges76 silver badges82 bronze badges

answered Mar 3, 2021 at 10:12

Minisha

2,2873 gold badges34 silver badges59 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Spark, how to read a file as byte array?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related