c++: working with bytes

Question

My problem is, that I need to load a binary file and work with single bits from the file. After that I need to save it out as bytes of course.

My main problem is - what datatype to choose to work in - char or long int? Can I somehow work with chars?

Btw, how long is your file? Is it really necessary to think about optimalization already? And do you have to change single bytes or are the 'single bits' chunks of bytes? — Michel Keijzers
– Michel Keijzers, Commented Mar 9, 2012 at 14:30
@Deepak: Using ints to parse binary data is just asking for endianness problems. — KillianDS
– KillianDS, Commented Mar 9, 2012 at 14:30
It depends on what operations he wants to do, ANDing 8 chars is equal to one int operation.(x64) — Deepak
– Deepak, Commented Mar 9, 2012 at 14:34
Deepak: sizeof(long int) is not always the same as sizeof(int). It's certainly not on the setup I'm typing this on. — Peter
– Peter, Commented Mar 9, 2012 at 14:38
@Deepak: when its the same, then why sizefo(long int) != sizeof(int) here? — PlasmaHH
– PlasmaHH, Commented Mar 9, 2012 at 14:41

Steve Townsend · Accepted Answer · 2012-03-09 14:45:13Z

6

Unless performance is mission-critical here, use whatever makes your code easiest to understand and maintain.

answered Mar 9, 2012 at 14:45

Steve Townsend

54.4k9 gold badges100 silver badges145 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

daramarak Over a year ago

Disregard my answer, this is rule #1

KillianDS Over a year ago

+1 And do not reinvent the wheel if possible, if you do not have to work with a predefined serialization format, don't go invent one.

daramarak Over a year ago

Agree, even though it is so fun reinventing the wheel. "Look, mine is squared"

Steve Townsend Over a year ago

It's possible that a clarified question could invite a more detailed recommendation. Not clear to me that this needs to be overthought from the info to hand, though.

Community · Accepted Answer · 2017-05-23 12:03:13Z

5

Before beginning to code any thing make sure you understand endianess, c++ type sizes, and how strange they might be.

The unsigned char is the only type that is a fixed size (natural byte of the machine, normally 8 bits). So if you design for portability that is a safe bet. But it isn't hard to just use the unsigned int or even a long long to speed up the process and use size_of to find out how many bits you are getting in each read, although the code gets more complex that way.

You should know that for true portability none of the internal types of c++ is fixed. An unsigned char might have 9 bits, and the int might be as small as in the range of 0 to 65535, as noted in this and this answer

Another alternative, as user1200129 suggests, is to use the boost integer library to reduce all these uncertainties. This is if you have boost available on your platform. Although if going for external libraries there are many serializing libraries to choose from.

But first and foremost before even start optimizing, make something simple that work. Then you can start profiling when you start experiencing timing issues.

edited May 23, 2017 at 12:03

CommunityBot

11 silver badge

answered Mar 9, 2012 at 14:46

daramarak

6,1851 gold badge34 silver badges51 bronze badges

2 Comments

daramarak Over a year ago

Yeah, world of programming gets strange at once you start exploring alien platforms ;)

01100110 Over a year ago

You can use boost integer.hpp for portable int types. For example, if you need to ensure you get 64 signed bits, you can use boost::int64_t across different compilers and operating systems and you'll always get the type you expect. This is especially important when you need to reinterpret_cast data.

PearsonArtPhoto · Accepted Answer · 2012-03-09 14:28:46Z

3

It really just depends on what you are wanting to do, but I would say in general, the best speed will be to stick with the size of integers that your program is compiled in. So if you have a 32 bit program, then choose 32 bit integers, and if you have 64 bit, choose 64 bit.

This could be different if there are some bytes in your file, or if there are integers. Without knowing the exact structure of your file, it's difficult to determine what the optimal value is.

answered Mar 9, 2012 at 14:28

PearsonArtPhoto

39.9k17 gold badges116 silver badges147 bronze badges

Comments

Michel Keijzers · Accepted Answer · 2012-03-09 14:34:12Z

1

Your sentences are not really correct English, but as far as I can interpret the question you can beter use unsigned char (which is a byte) type to be able to modify each byte separately.

Edit: changed according to comment.

edited Mar 9, 2012 at 14:34

answered Mar 9, 2012 at 14:26

Michel Keijzers

15.4k28 gold badges100 silver badges124 bronze badges

6 Comments

MByD Over a year ago

What's an unsigned byte? byte is an unsigned char.

Prof. Falken Over a year ago

Now it is somewhat proper English. :)

Mr Lister Over a year ago

Since there is no definition for byte in C, you can't say if it's signed or not.

Mr Lister Over a year ago

@Michel you edited it the wrong way round. you were looking for unsigned char.

Michel Keijzers Over a year ago

Fixed (Friday Afternoon Syndrome)

|

111111 · Accepted Answer · 2012-03-09 14:45:02Z

1

If you are dealing with bytes then the best way to do this is to use a size specific type.

#include <algorithm>
#include <iterator>
#include <cinttypes>
#include <vector>
#include <fstream>

int main()
{
     std::vector<int8_t> file_data;
     std::ifstream file("file_name", std::ios::binary);

     //read
     std::copy(std::istream_iterator<int8_t>(file),
               std::istream_iterator<int8_t>(),
               std::back_inserter(file_data));

     //write
     std::ofstream out("outfile");           
     std::copy(file_data.begin(), file_data.end(),
               std::ostream_iterator<int8_t>(out));

}

EDIT fixed bug

edited Mar 9, 2012 at 14:45

answered Mar 9, 2012 at 14:39

111111

16.3k6 gold badges49 silver badges64 bronze badges

2 Comments

daramarak Over a year ago

the uint8_t are not guaranteed to be defined for all systems. But it much more clearly states the intent of the use.

Mike DeSimone Over a year ago

The C99 standard has been around a long time, and almost all systems have <stdint.h>. (I can't think of one that doesn't, honestly. It's one of the easiest headers ever to provide.) The C++ equivalent might not be there, but that's easily worked around.

Mike DeSimone · Accepted Answer · 2012-03-09 14:54:36Z

If you need to enforce how many bits are in an integer type, you need to be using the <stdint.h> header. It is present in both C and C++. It defines type such as uint8_t (8-bit unsigned integer), which are guaranteed to resolve to the proper type on the platform. It also tells other programmers who read your code that the number of bits is important.

If you're worrying about performance, you might want to use the larger-than-8-bits types, such as uint32_t. However, when reading and writing files, you will need to pay attention to the endianess of your system. Notably, if you have a little-endian system (e.g. x86, most all ARM), then the 32-bit value 0x12345678 will be written to the file as the four bytes 0x78 0x56 0x34 0x12, while if you have a big-endian system (e.g. Sparc, PowerPC, Cell, some ARM, and the Internet), it will be written as 0x12 0x34 0x56 0x78. (same goes or reading). You can, of course, work with 8-bit types and avoid this issue entirely.

Collectives™ on Stack Overflow

c++: working with bytes

6 Answers 6

4 Comments

2 Comments

Comments

6 Comments

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

4 Comments

2 Comments

Comments

6 Comments

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related