Google protocol buffers and use of std::string for arbitrary binary data

Question

Related Question: vector <unsigned char> vs string for binary data.

My code uses vector<unsigned char> for arbitrary binary data. However, a lot of my code has to interface to Google's protocol buffers code. Protocol buffers uses std::string for arbitrary binary data. This makes for a lot of ugly allocate/copy/free cycles just to move data between Google protocol buffers and my code. It also makes for a lot of cases where I need two constructors (one which takes a vector and one a string) or two functions to convert a function to binary wire format.

The code deals with raw structures a lot internally because structures are content-addressable (stored and retrieved by hash), signed, and so on. So it's not just a matter of the interface to Google's protocol buffers. Objects are handled in raw forms in other parts of the code as well.

One thing I could do is just cut all my code over to use std::string for arbitrary binary data. Another thing I could do is try to work out more efficient ways to store and retrieve my vectors into Google protocol buffer objects. I guess my other choice would be to create standard, simple, but slow conversion functions to strings and always use them. This would avoid the rampant code duplication, but would be worst from a performance standpoint.

Any suggestions? Any better choices I'm missing?

This is what I'm trying to avoid:

if(SomeCase)
{
    std::vector<unsigned char> rawObject(objectdata().size());
    memcpy(&rawObject.front(), objectdata().data(), objectdata().size());
    DoSometingWith(rawObject);
}

The allocate, copy, process, free is completely senseless when the raw data is already sitting there.

Presumably the "a lot of ugly allocate/copy/free cycles" comment is based on measurements telling you that dynamic memory allocation is eating up cycles like mad. In that case, consider a small objects allocator. There is one in the Loki library. — Cheers and hth. - Alf
– Cheers and hth. - Alf, Commented Feb 21, 2012 at 6:50
@Alf: Honestly, right now it's more the ugliness of the code that bothers me. (Allocate a vector, call memcpy, use the vector, destroy the vector, when the data was already sitting in memory.) See the update. — David Schwartz
– David Schwartz, Commented Feb 21, 2012 at 6:56
Why memcpy? Why not std::vector<unsigned char> rawObject(objectdata().data(), objectdata().data() + objectdata().size()); ? — Sarfaraz Nawaz
– Sarfaraz Nawaz, Commented Feb 21, 2012 at 7:09

Matthieu M. · Accepted Answer · 2012-02-21 07:20:48Z

4

There are two ways to avoid copying that I know of and have seen in use.

The traditional way is indeed to pass a pointer/reference to a known entity. While this works fine and with a minimum of fuss, the issue is that it ties you up to a given representation, which entails conversions (as you experienced) when necessary.

The other way I discovered with LLVM:

The idea is amazingly simple: both hold a T* pointing to the start of an array of T and a size_t indicating the number of elements.

What is magical is that they completely hide the actual storage, be it a string, a vector, a dynamically or statically allocated C-array... it does not matter. The interface presented is completely uniform and no copy is involved.

The only caveat is that they do not take ownership of the memory (Ref!) so subtle bugs might creep in if you do not take care. Still, it is usually fine if you only use them in transient operations (within a function, for example) and do not store them for later use.

I have found them incredibly handy in buffer manipulations, especially thanks to the free slicing operations. Ranges are just so much easier to manipulate than pairs of iterators.

There is also a third way I have experienced, but never used in serious code up until now. The idea is that a vector<unsigned char> is a very low-level representation. By raising the abstraction layer and use, say, a Buffer class, you can completely encapsulate the exact way the memory is stored so that it becomes a non-issue, as far as your code is concerned.

And then, feel free to choose the one memory representation that requires the less conversion.

answered Feb 21, 2012 at 7:20

Matthieu M.

303k58 gold badges495 silver badges754 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

David Schwartz Over a year ago

That is definitely a great option. I can create a class that takes a const void * and a size_t. I can construct them easily, on the stack, from either a vector or a string. Likely, the optimizer will even see through it if the class 100% inlines. (I can always use them only for the duration of a function call, made by the owner of the object wrapped.)

Matthieu M. Over a year ago

@DavidSchwartz: why const void* ? T const* seems much better, at the cost of a reinterpret_cast if you have this char/unsigned char issue.

David Schwartz Over a year ago

The whole point is to wrap binary data that has no particular structure. void * seems most natural to me. This code is kind of what I was thinking.

Sarfaraz Nawaz Over a year ago

@DavidSchwartz: I don't understand why you're using reinterpret_cast in your code when static_cast is sufficient?

David Schwartz Over a year ago

@Nawaz Because that's what Google's protocol buffers does to cast between const char * and const void *. I'm not really sure why they do though. (I agree that static_cast should be used where, as here, it is sufficient.)

Cheers and hth. - Alf · Accepted Answer · 2012-02-21 07:11:28Z

1

To avoid this code (which you present),

if(SomeCase)
{
    std::vector<unsigned char> rawObject(objectdata().size());
    memcpy(&rawObject.front(), objectdata().data(), objectdata().size());
    DoSometingWith(rawObject);
}

where presumably objectData is a std::string, consider

typedef unsigned char      Byte;
typedef std::vector<Byte>  ByteVector;

and then e.g.

if( someCase )
{
    auto const& s = objectData;
    doSomethingWith( ByteVector( s.begin(), s.end() ) );
}

edited Feb 21, 2012 at 7:11

answered Feb 21, 2012 at 7:06

Cheers and hth. - Alf

146k15 gold badges218 silver badges342 bronze badges

2 Comments

David Schwartz Over a year ago

That looks a lot less ugly. It still is ugly in the sense that it still has an allocate/copy/free for no reason.

André Caron Over a year ago

This is syntaxically much nicer, but I've had the same problem as OP when I was working with Qt. It's annoying to have to convert to and from std::string for every message on the network.

Collectives™ on Stack Overflow

Google protocol buffers and use of std::string for arbitrary binary data

2 Answers 2

5 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related