I’m on a quest of figuring out if it’s possible to write an endianness independent code for reading from binary streams with the following qualities:
- Does not depend on host’s endianness (i.e. the code should not try to determine local endianness via
ifdef
s or some other means). Obviously, endianness of source data has to be known. - Preferably the code should be C17 standard compliant, but widely adopted implementation defined behavior is good enough. No type punning, no violations of strict aliasing and pointer alignment rules.
- Modern compilers should produce reasonably efficient instructions, somewhat close to
mov
ormov
+bswap
. - Integer types representation can be assumed to be 2’s complement.
- Floating point numbers are IEEE 754 compliant and have the same endianness rules as integers.
- Mixed endianness and platforms with flexible runtime endianness can be ignored.
To simplify things, I’ll only consider 32-bit types.
Integers.
int32_t read_i32_from_le(uint8_t* data)
{
int32_t r = ((uint32_t)data[0] << 0) |
((uint32_t)data[1] << 8) |
((uint32_t)data[2] << 16) |
((uint32_t)data[3] << 24);
return r;
}
uint32_t read_u32_from_le(uint8_t* data)
{
uint32_t r = ((uint32_t)data[0] << 0) |
((uint32_t)data[1] << 8) |
((uint32_t)data[2] << 16) |
((uint32_t)data[3] << 24);
return r;
}
int32_t read_i32_from_be(uint8_t* data)
{
int32_t r = ((uint32_t)data[3] << 0) |
((uint32_t)data[2] << 8) |
((uint32_t)data[1] << 16) |
((uint32_t)data[0] << 24);
return r;
}
uint32_t read_u32_from_be(uint8_t* data)
{
uint32_t r = ((uint32_t)data[3] << 0) |
((uint32_t)data[2] << 8) |
((uint32_t)data[1] << 16) |
((uint32_t)data[0] << 24);
return r;
}
This code mostly conforms to the aforementioned rules. It does contain implementation defined behavior (assignment from uint32_t
to int32_t
), but, as stated previously, it’s acceptable.
Unfortunately, it fails on the efficiency side of things. Or more specifically, it’s MSVC that does the failing (godbolt):
read_i32_from_le PROC ; COMDAT
movzx edx, BYTE PTR [rcx+2]
movzx eax, BYTE PTR [rcx+3]
shl eax, 8
or eax, edx
movzx edx, BYTE PTR [rcx+1]
movzx ecx, BYTE PTR [rcx]
shl eax, 8
or eax, edx
shl eax, 8
or eax, ecx
ret 0
Floats.
To read floats in endianness independent manner we can reuse read_u32_*
routines:
float read_f32_from_le(uint8_t* data)
{
uint32_t bits = read_u32_from_le(data);
float r;
memcpy(&r, &bits, 4);
return r;
}
Unsurprisingly, MSVC still struggles to generate reasonable instructions (godbolt):
read_f32_from_le PROC ; COMDAT
movzx eax, BYTE PTR [rcx+2]
movzx edx, BYTE PTR [rcx+3]
shl edx, 8
or edx, eax
movzx eax, BYTE PTR [rcx+1]
shl edx, 8
or edx, eax
movzx eax, BYTE PTR [rcx]
shl edx, 8
or edx, eax
mov DWORD PTR r$[rsp], edx
movss xmm0, DWORD PTR r$[rsp]
ret 0
Questions.
- Is there some other way to achieve the desired result?
- Is this a compiler bug, and should I file it with MSVC?
Pavel T. is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.