ByteStringCoder for reading data in Apache Beam

Question

I am trying to read data with version 2.4.0 of Apache Beam using the standard TextIO.read(). The data has to be read as a ByteString.

Unfortunately, it doesn't seem like Apache Beam supports .withCoder() in the same way Dataflow does. I can't seem to find an alternative way to introduce a coder. Furthermore, it seems like ByteStringCoder is no longer included in the coders of Apache Beam.

What's the best way to get the same result of Dataflow's .withCoder(ByteStringCoder.of()) with the latest version of Apache Beam? Coders are still present in Apache Beam so there has to be some way to use them.

Marek Šimůnek · Accepted Answer · 2018-11-25 19:37:42Z

1

ByteStringCoder is located in beam-sdks-java-extensions-protobuf module so you need to include dependency

https://mvnrepository.com/artifact/org.apache.beam/beam-sdks-java-extensions-protobuf

As for the TextIO: It uses StringUtf8Coder so you probably need to write your own BoundedSource/UnboundedSource. And then use:

pipeline.apply(Read.from(yourCreatedSource))

You could take inspiration of current TextSource, where you can probably only change type String of FileBasedSource for ByteString.

https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextSource.java

answered Nov 25, 2018 at 19:37

Marek Šimůnek

863 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

ByteStringCoder for reading data in Apache Beam

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related