Problem when copying byte array into c structure

Question

I know this has probably been answered before, but I still can’t wrap my head around a solution to what I believe to be an endianness problem. I built a quick example below which demonstrates my test code.

https://onlinegdb.com/SJtEatMvS

In this example, I have a simple byte array. In reality, this byte array is a larger data set collected over CAN, but for the sake of this question I’ve used a smaller hard coded array.

Objective

My goal, in c, is to copy the byte array into a structure, preserving the order that the array was in (if that makes sense). For example

The data set contains:

{0x12, 0x34, 0x56, 0x78, 0x0A, 0x06, 0x77}

And the structure definition is

typedef struct {
    uint8_t  test0;
    uint16_t test1;
    uint32_t test2;
} Foo_t;

I would want 0x12 copied into test0, {0x3456} copied into test1, and {0x780A0677} copied into test2. As mentioned above, I used a small array for testing, but the actual array is quite large, so assigning the struct members manually is not an option for me.

I know memcpy is not the problem as it doesn’t care about endianness, and the actual problem is my assumptions about how the data should be aligned. In terms of host, this is running on a windows system, which I believe is little endian.

You have two problems if you want to use memcpy: endianness is one, as you suspect you'll have to swap the byte order for multi-byte values on Windows. The other problem is alignment/padding: by default, your struct above will have padding bytes to align each field on a natural boundary (an offset divisible by its size). — 500 - Internal Server Error
– 500 - Internal Server Error, Commented Sep 20, 2019 at 17:46
Flip the order of the struct members, do not let the compiler pad, then copy the bytes from last to beginning. That's one option. — Bayleef
– Bayleef, Commented Sep 20, 2019 at 19:14
Due to the issues already mentioned, you'd be better off just honestly writing code to deserialize the byte array data and assign to struct fields one by one, taking care of endianness as necessary. — hyde
– hyde, Commented Sep 20, 2019 at 19:38

Mentat · Accepted Answer · 2019-09-21 18:27:22Z

1

I remove my original answer due to not fulling understand your problem. I get it now after reading the follow article on: Writing endian-independent code in C

First the alignment issue:

As mention by 500 - Internal Server Error

You will have issues coping the data because your structure will contain padding. In your example, 1 byte will be added to the structure.

Here is a memory layout example obtain from VS on a 32 bit C implementation.

size = 8
Address of test0        = 5504200
Padding added here at address 5504201
Address of test1        = 5504202
Address of test2        = 5504204

To specify the alignment rules the compiler should use, use the preprocessor directive pack.

// Aligns on byte boundaries, then restore alignment value to system defaults
#pragma pack ( 1 )
#pragma pack ()

// Aligns on byte boundaries, restores previously assigned alignment value.
#pragma pack ( push, 1 )
#pragma pack (pop)

Using your example, the structure definition will look something like this:

#pragma pack ( 1 )
typedef struct {
    unsigned char  test0;
    unsigned short test1;
    unsigned int   test2;
} Foo_t;
#pragma pack ()

Foo_t s2;

printf("\nsize = %d\n", sizeof(Foo_t));

printf("   Address of test0        = %u\n", &s2.test0);
printf("   Address of test1        = %u\n", &s2.test1);
printf("   Address of test2        = %u\n", &s2.test2);

Result:

size = 7
Address of test0        = 10287904
Address of test1        = 10287905
Address of test2        = 10287907

Second the endian issue:

The issue here is how integers are being stored in memory on a 32-bit x86 machines. On x86 machines they are store in little endian order.

For an example, copying a 2 byte array containing the bytes x34 & x56 into an short integer will be stored as x56 (low order byte) x34 (next byte). Which is not what you wanted.

To fix this problem you need to switch the bytes around as other suggested. My take on this is to create a function that would do the byte swap in place.

Example:

int main()
{

#pragma pack ( 1 )
typedef struct {
    unsigned char  test0;
    unsigned short test1;
    unsigned int   test2;
} Foo_t;
#pragma pack ()

    unsigned char tempBuf[7] = { 0x12, 0x34, 0x56, 0x78, 0x0A, 0x06, 0x77 };

    Foo_t foo;

    memcpy(&foo, &tempBuf[0], 7);

    //foo.test0 = netToHost (&foo,0,1);  // not needed
    foo.test1 = reverseByteOrder(&foo, 1, 2);
    foo.test2 = reverseByteOrder(&foo, 3, 4);

    printf("\n After memcpy We have %02X %04X %08X\n", foo.test0, foo.test1, foo.test2);
}


int reverseByteOrder(char array[], int startIndex, int size)
{
    int intNumber =0;

    for (int i = 0; i < size; i++)
        intNumber = (intNumber << 8) | array[startIndex + i];

    return intNumber;
}

The output is:

After memcpy We have 12 3456 780A0677

edited Sep 21, 2019 at 18:27

answered Sep 20, 2019 at 19:33

Mentat

3811 silver badge5 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Matthieu Over a year ago

It's better to use #pragma pack(push,1) / #pragma(pop) (if available on your compiler) to retain a potentially-previously-set pack value.

Mentat Over a year ago

Thanks Matthieu for both the edits and the additional pack information. I have included your comments in the answer.

Matthieu Over a year ago

Also, note that you have functions ntoh*() that perform the byte reordering for you, if it's needed.

Birts Over a year ago

Thanks to both of you. This explanation was extremely helpful!

Matthieu · Accepted Answer · 2019-09-20 20:15:16Z

1

Endianness is linked to a CPU, not an OS. But as Windows is running on x86 only and x86 is little-endian, Windows is little endian (well, they appear to have an ARM version too, but most ARM are little-endian as well).

As your data is big-endian, you'll have to convert it to your processor endianness. But big-endian is also the standard network byte order so you can rely on ntoh*() functions to do that for you. Unfortunately it means you'll have to do it manually for every field...

If your CPU was big endianned, you could have packed your struct with #pragma pack(1) and memcpy() it (or cast a pointer).

answered Sep 20, 2019 at 20:15

Matthieu

3,1456 gold badges63 silver badges90 bronze badges

Comments

Wayne Taylor · Accepted Answer · 2019-09-20 20:23:12Z

uint8_t data[] = {0x12, 0x34, 0x56, 0x78, 0x0A, 0x06, 0x77};
typedef struct {
    uint8_t  test0;
    uint16_t test1;
    uint32_t test2;
} Foo_t;

Foo_t fs;
fs.test0 = data[0];
fs.test1 = data[1]<<8 + data[2];
fs.test2 = data[3]<<24 + data[4]<<16 + data[5]<<8 + data[6];

If your processor was big endian, you could cheat a little..

fs.test0 = data[0];
fs.test1 = *(uint16_t*)&data[1];
fs.text2 = *(uint32_t*)&data[3];

If your array has the bytes in an order that match your processor and your struct has many variables in it you could use __packed attribute and a memcpy().

typedef __packed struct {
    uint8_t  test0;
    uint16_t test1;
    uint32_t test2;
} Foo_t;

Foo_t fs;
memcpy(&fs, data, sizeof(fs));

Collectives™ on Stack Overflow

Problem when copying byte array into c structure

Objective

3 Answers 3

4 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

Objective

3 Answers 3

4 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related