3

I have a lump of binary data in the form of const std::vector<unsigned char>, and want to be able to extract individual fields from that, such as 4 bytes for an integer, 1 for a boolean, etc. This needs to be, as far as possible, both efficient and simple. eg. It should be able to read the data in place without needing to copy it (eg. into a string or array). And it should be able to read one field at a time, like a parser, since the lump of data does not have a fixed format. I already know how to determine what type of field to read in each case - the problem is getting a usable interface on top of an std::vector for doing this.

However I can't find a simple way to get this data into an easily usable form that gives me useful read functionality. eg. std::basic_istringstream<unsigned char> gives me a reading interface, but it seems like I need to copy the data into a temporary std::basic_string<unsigned char> first, which is not idea for bigger blocks of data.

Maybe there is some way I can use a streambuf in this situation to read the data in place, but it would appear that I'd need to derive my own streambuf class to do that.

It occurs to me that I can probably just use sscanf on the vector's data(), and that would seem to be both more succinct and more efficient than the C++ standard library alternatives. EDIT: Having been reminded that sscanf doesn't do what I wrongly thought it did, I actually don't know a clean way to do this in C or C++. But am I missing something, and if so, what?

Improve this question
5
  • You could just use a std::string and it's data() method. You could use your bitwise operators without the need to copy anything. Commented May 5, 2011 at 18:24
  • The data doesn't arrive in the form of a std::string. And I'm not sure which bitwise operators you're referring to - I need to read bytes sequentially. Commented May 5, 2011 at 18:26
  • I am confused by your conflicting requirements. You say that you have "a lump of binary data", but you say that you can "probably just use sscanf." scanf reads text-formatted data, not binary-formatted data. Let me put it this way: if your vector has an int, it is stored as a four-byte 2's-complement array of 32 bits, or is stored as several characters, each character in the range 0-9? Commented May 5, 2011 at 19:28
  • It's not a conflicting requirement, just me forgetting what sscanf actually does! I'll remove that from the question: thanks for bringing it up. My data is generally not stored in plain text. Commented May 5, 2011 at 19:32
  • Similarly, I don't think having an istream (or its derivitive) would do you any good. operator<<(istream&, T) is for reading from formatted text, not unformatted binary data. See the Reader class in my post. Commented May 5, 2011 at 19:44

5 Answers 5

Reset to default
4

You have access to the data in a vector through its operator[]. A vector's data is guranteed to be stored in a single contiguous array, and [] returns a reference to a member of that array. You may use that reference directly, or through a memcpy.

std::vector<unsigned char> v;
...
byteField = v[12];
memcpy(&intField, &v[13], sizeof intField);
memcpy(charArray, &v[20], lengthOfCharArray);

EDIT 1: If you want something "more convenient" that that, you could try:

template <class T>
ReadFromVector(T& t, std::size_t offset, 
  const std::vector<unsigned char>& v) {
  memcpy(&t, &v[offset], sizeof(T));
}

Usage would be:

std::vector<unsigned char> v;
...
char c;
int i;
uint64_t ull;
ReadFromVector(c, 17, v);
ReadFromVector(i, 99, v);
ReadFromVector(ull, 43, v);

EDIT 2:

struct Reader {
  const std::vector<unsigned char>& v;
  std::size_t offset;
  Reader(const std::vector<unsigned char>& v) : v(v), offset() {}
  template <class T>
  Reader& operator>>(T&t) {
    memcpy(&t, &v[offset], sizeof t);
    offset += sizeof t;
    return *this;
  }
  void operator+=(int i) { offset += i };
  char *getStringPointer() { return &v[offset]; }
};

Usage:

std::vector<unsigned char> v;
Reader r(v);
int i; uint64_t ull;
r >> i >> ull;
char *companyName = r.getStringPointer();
r += strlen(companyName);
Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

I really need a more convenient stream-style interface because the fields I need to read from the data is not a simple fixed format like that.
Are the fields tightly packed, and you are guaranteed to read them in order?
Yes, and yes. It would appear that some variation on your edited suggestion is probably the way to go, but I am surprised that there appears to be no simple way to treat a vector as a backing store for a stream or streambuf.
@Kylotan, You can use the a vector as a backing store for a stream or streambuf. But having a stream won't do you any good. The stream operators are all formatted operators (for example, they change the string "1234" to an int with value 1234). You are searching for unformatted input, which is a different kettle of fish.
streambufs feature a couple of unformatted read functions, which would help a little (since it would manage the read position for me). It's not as useful as the .NET BinaryReader or something similar, of course - that would be ideal.
|
1

If your vector stores binary data, you can't use sscanf or similar, they work on text. For converting a byte for a bool is simple enough

bool b = my_vec[10];

For extracting an unsigned int that's stored in big endian order (assuming your ints are 32 bits):

unsigned int i = my_vec[10] << 24 | my_vec[11] << 16 | my_vec[12] << 8 | my_vec[13];

A 16 bit unsigned short would be similar:

 unsigned short s = my_vec[10] << 8 | my_vec[11];¨
Improve this answer

2 Comments

I'm sorry, perhaps my original question wasn't clear. I don't have a problem extracting individual bytes. But what I need is a more useful stream-style interface for doing this because I have arbitrary amounts of data. I'll edit the question to be clearer.
pack the code above in read_uint32(...), a read_bool(...) a read_blob() etc. functions that extracts the various elements, and remember your current read position.
1

If you can afford the Qt dependency, QByteArray has the fromRawData() named constructor, which wraps existing data buffers in a QByteArray without copying the data. With that byte array, you can the feed a QTextStream.

I'm not aware of any such function in the standard streams library (short of implementing your own streambuf, of course), but I'd love to be proved wrong :)

Improve this answer

1 Comment

Sadly Qt is not an option for me, but I've upvoted this as I expect this would be useful for others to know if they come across this question.
1

You can use a struct that describes the data you are trying to extract. You can move data from your vector into the struct like this:

struct MyData {
    int intVal;
    bool boolVal;
    char[15] stringVal;
} __attribute__((__packed__));

// assuming all extracted types are prefixed with a one byte indicator.
// Also assumes "vec" is your populated vector
int pos = 0;
while (pos < vec.size()-1) {
    switch(vec[pos++]) {
        case 0: { // handle int
            int intValue; 
            memcpy(&vec[pos], &intValue, sizeof(int));
            pos += sizeof(int); 
            // do something with handled value
            break;
        }
        case 1: { // handle double
            double doubleValue; 
            memcpy(&vec[pos], &doubleValue, sizeof(double));
            pos += sizeof(double); 
            // do something with handled value
            break;
        }
        case 2: { // handle MyData
            struct MyData data; 
            memcpy(&vec[pos], &data, sizeof(struct MyData));
            pos += sizeof(struct MyData); 
            // do something with handled value
            break;
        }
        default: {
            // ERROR: unknown type indicator
            break;
        }
    }
}
Improve this answer

11 Comments

Remember that the binary data must be stored in this format when added, and you must be careful of struct packing. I would recommend atleast using fixed-width types defined in stdint.h or even better using compiler flags to stop struct packing.
Unfortunately the data is arbitrary and often of variable length so that approach won't work for me.
arbitrary and variable length... how exactly are you able to parse binary data if you don't know the incoming format? I'm probably just missing something.
I know the meta-format, the same way as you'd read a web page. You don't know how many of each tag you're going to get or how they'll be arranged but you know each will start with a < and finish with a >.
Right, but in your case, what is the "start tag" and how does it indicate to you how to handle the binary data?
|
0

Use a for loop to iterate over the vector and use bitwise operators to access each bit group. For example, to access the upper four bits of the first usigned char in your vector:

int myInt = vec[0] & 0xF0;

To read the fifth bit from the right, right after the chunk we just read:

bool myBool = vec[0] & 0x08;

The three least significant (lowest) bits can be accesed like so:

int myInt2 = vec[0] & 0x07;

You can then repeat this process (using a for loop) for every element in your vector.

Improve this answer

1 Comment

I didn't mention anything about individual bits in my question.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.