How to encode protocol buffer string to binary using protoc

Question

I been trying to encode strings using protoc cli utility. Noticed that output still contains plain text. What am i doing wrong?

osboxes@osboxes:~/proto/bin$ cat ./teststring.proto
syntax = "proto2";
message Test2 {
  optional string b = 2;
}

echo b:\"my_testing_string\"|./protoc --encode Test2 teststring.proto>result.out

result.out contains:

^R^Qmy_testing_string

protoc versions libprotoc 3.6.0 and libprotoc 2.5.0

Are you sure it isn't working fine? Displaying it to the console is bound to cause problems - the console is text, not binary. But pipe it to a file, and it'll probably be right. You can test at protogen.marcgravell.com/decode - just upload your test file there and see what it makes of it — Marc Gravell
– Marc Gravell, Commented Jun 28, 2018 at 18:21
@MarcGravell I think it was exactly what i did in the example above... Piping the output of encoding to the file result.out — kpavel
– kpavel, Commented Jun 28, 2018 at 18:26
Ok, my mistake. Now: you're displaying text - what are the hex of that file? Looking at it as text is doomed to failure. Note that since protobuf encodes strings as utf8, it is expected that your text appears "as is". What is interesting to me here is the first 6 (or so) bytes of the file. As hex, not as characters. — Marc Gravell
– Marc Gravell, Commented Jun 28, 2018 at 18:28
Note - I think the decode page above will display the hex if you upload a file — Marc Gravell
– Marc Gravell, Commented Jun 28, 2018 at 18:29
@MarcGravell Thanks! You right. The decode page above indeed shows expected hex. — kpavel
– kpavel, Commented Jun 28, 2018 at 19:09

Marc Gravell · Accepted Answer · 2018-06-28 23:31:18Z

3

Just to formalize in an answer:

The command as written should be fine; the output is protobuf binary - it just resembles text because protobuf uses utf-8 to encode strings, and your content is dominated by a string. However, despite this: the file isn't actually text, and you should usually use a hex viewer or similar if you need to inspect it.

If you want to understand the internals of a file, https://protogen.marcgravell.com/decode is a good resource - it rips an input file or hex string following the protocol rules, and tells you what each byte means (field headers, length prefixes, payloads, etc).

I'm guessing your file is actually:

(hex) 10 11 6D 79 5F etc

i.e. 0x10 = "field 2, length prefixed", 0x11 = 17 (the payload length, encoded as varint), then "my_testing_string" encoded as 17 bytes of UTF8.

edited Jun 28, 2018 at 23:31

answered Jun 28, 2018 at 23:23

Marc Gravell

1.1m273 gold badges2.6k silver badges3k bronze badges

Sign up to request clarification or add additional context in comments.

Comments

bb7bb · Accepted Answer · 2019-11-12 11:01:29Z

3

protoc --proto_path=${protobuf_path} --encode=${protobuf_message} ${protobuf_file} < ${source_file} > ${output_file}

and in this case:

protoc --proto_path=~/proto/bin --encode="Test2" ~/proto/bin/teststring.proto < ${source.txt} > ./output.bin

or:

cat b:\"my_testing_string\" | protoc --proto_path=~/proto/bin --encode="Test2" ~/proto/bin/teststring.proto > ./output.bin

edited Nov 12, 2019 at 11:01

answered Nov 12, 2019 at 5:28

bb7bb

313 bronze badges

1 Comment

Benjamin Loison Over a year ago

You mean echo b:\"my_testing_string\" not cat b:\"my_testing_string\", right?

Collectives™ on Stack Overflow

How to encode protocol buffer string to binary using protoc

2 Answers 2

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related