Wrong endianness of bit-sized integers

[KOLANICH]

For now bit-sized integers are always big-endian. I think that endianness also should affect them.

[JaapAap]

FYI, I have actually implemented big-endian, little endian, signed and unsigned bit fields (compiler and C++ runtime). I am currently testing my solution. When I am happy with the results I will generate a pull request for these.

[GreyCat]

The main problem here is that other implementations beside "big-endian" are not very trivial. For example, consider this:

x[0]     x[1]     x[2]     
76543210 76543210 76543210
abbbbbbb bbbbbbbb bccccccc

Intepreting "b" as big-endian is straightforward:

b =
  ((x[2] & 0b1000_0000) >> 7) |
  (x[1] << 1) |
  ((x[0] & 0b0111_1111) << 9)

But I see at least two possible interpretations of "b" as little-endian here:

// Method 1
// Dividing by x byte boundaries, reassembling by them as well
b =
  ((x[0] & 0b0111_1111)) |
  (x[1] << 7) |
  ((x[2] & 0b1000_0000) << 8) // actually >> 7 << 15

// Method 2
// First, reassemble "big-endian" 2-byte integer
b0 =
  ((x[2] & 0b1000_0000) >> 7) |
  (x[1] << 1) |
  ((x[0] & 0b0111_1111) << 9)
// Second, reverse endianness in it, i.e. reassemble per 2-byte integer byte boundaries
b =
  ((b0 & 0xff00) >> 8) |
  ((b0 & 0x00ff) << 8)

For example, for a given x = [0x4a, 0x9d, 0xd0]:

• first method yields b = 0xceca = 0b1100_1110_1100_1010
• second method yields b = 0x3b95 = 0b0011_1011_1001_0101

And this is even without touching a subject of bit order in bytes — which is also actually important, as many compression stream formats rely on that one.

[JaapAap]

You are right that this is tricky - I have implemented neither of those cases as the LE vs BE in my use case relate to the byte order. What I have currently implemented is the cases in which bytes are ordered from LS to MS, bits within bytes are ordered from MS to LS. The endianness is therefore related to the byte order only. That means that in your example in the LE case the bits 0-23 in the stream are assumed to be in the following order:

              x[0]                              x[1]                              x[2] 
		
7   6   5   4   3   2   1   0     7   6   5   4   3   2   1   0     7   6   5   4   3   2   1   0
b7  b6  b5  b4  b3  b2  b1  b0    b15 b14 b13 b12 b11 b10 b9 b8     b23 b22 b21 b20 b19 b18 b17 b16

so selecting bits 1-16 into a 16-bit integer (bits i15-i14-...-i0 from MSB to LSB) in my case gives the following mapping:

              x[0]                              x[1]                              x[2]     
7   6   5   4   3   2   1   0     7   6   5   4   3   2   1   0     7   6   5   4   3   2   1   0
i6  i5  i4  i3  i2  i1  i0  X     i14 i13 i12 i11 i10 i9  i8  i7    X   X   X   X   X   X   X   i15

Unlike your example, in this case the selected bits are actually no longer consecutive due to the byte ordering being different from the bit ordering.

[JaapAap]

For completeness, if I understand well, your first method above maps to this layout:

           x[0]                              x[1]                              x[2]     
7  6   5   4   3   2   1   0     7   6   5   4   3   2   1   0     7   6   5   4   3   2   1   0
a  i6  i5  i4  i3  i2  i1  i0    i14 i13 i12 i11 i10 i9  i8  i7    i15 c   c   c   c   c   c   c

while the second would correspond to:

           x[0]                              x[1]                              x[2]     
7  6   5   4   3   2   1   0    7   6   5   4   3   2   1   0     7   6   5   4   3   2   1   0
a  i7  i6  i5  i4  i3  i2  i1   i0  i15 i14 i13 i12 i11 i10 i9    i8  c   c   c   c   c   c   c

Are you aware of any formats that actually use these methods? In a third potential interpretation I could think of the byte boundaries playing no role at all - in this case, the bits of the big-endian integer are just taken in reverse order regardless of the byte they come from:

           x[0]                              x[1]                              x[2]     
7  6   5   4   3   2   1   0     7   6   5   4   3   2   1   0     7   6   5   4   3   2   1   0
a  i0  i1  i2  i3  i4  i5  i6    i7  i8  i9  i10 i11 i12 i13 i14   i15 c   c   c   c   c   c   c

[JaapAap]

While it would not be too hard to add different bit-within-byte orderings within my current bit-field implementation, I guess that this is something that also affects the current integer types and would rather have to be handled in a broader sense?

[tempelmann]

I understand that GreyCat is no fan of this but I propose it anyway as maybe others can add to the discussion:

Often, C bit fields are used to describe small values. Like this:

struct x {
  int16_t a : 12;
  int16_t b : 4;
};

The order of the bits is clearly defined, for both big and little endian.

If the ksy syntax would adopt this syntax somehow, avoiding the current bit stream syntax, it would make an uncomplicated and non-confusing solution for these kinds of bit fields.

The runtime support would need a new bit-read function for this, though. One that gets passed the full size of the value (in bytes or bits) from which to extract the bits, so that it can do this right for little-endian (for big endian, that value can be ignored, unless it wants to also throw an error if it's asked to read more bits than are available).

[GreyCat]

We don't really need full size of the value. Here's a drop-in replacement procedure for read_bits_int that works in "little-endian, low-to-high bit order" fashion:

  def read_bits_int_le(n)
    bits_needed = n - @bits_left
    if bits_needed > 0
      # 1 bit  => 1 byte
      # 8 bits => 1 byte
      # 9 bits => 2 bytes
      bytes_needed = ((bits_needed - 1) / 8) + 1
      buf = read_bytes(bytes_needed)
      buf.each_byte { |byte|
        @bits |= (byte << @bits_left)
        @bits_left += 8
      }
    end

    # raw mask with required number of 1s, starting from lowest bit
    mask = (1 << n) - 1
    # derive reading result
    res = @bits & mask
    # remove bottom bits that we've just read by shifting
    @bits >>= n
    @bits_left -= n

    res
  end

Testing it:

# Raw bytes: 21 f3
INPUT = [0x21, 0xf3].pack('C*')

# Little-endian, low-to-high byte order
# Expected output: 321 f
stream = Kaitai::Struct::Stream.new(INPUT)
printf "%x\n", stream.read_bits_int_le(12)
printf "%x\n", stream.read_bits_int_le(4)

# Big-endian, high-to-low byte order (traditional)
# Expected output: 21f 3
stream = Kaitai::Struct::Stream.new(INPUT)
printf "%x\n", stream.read_bits_int(12)
printf "%x\n", stream.read_bits_int(4)

It works — actual output matches expected.

[GreyCat]

Here's a side-by-side comparison of the difference between these two algorithms for a reference:

[tempelmann]

I disagree with your above results being correct.

Here's the C code:

struct T {
  int16_t a : 12;
  int16_t b : 4;
};

unsigned short i = 0x1234;
struct T *v = (struct T *) &i;

Now I can expect in both LE and BE that:

v->a == 0x0234;
v->b == 0x0001;

Your code doesn't meet that:

You're wrong where you say # Expected output: 21f 3 - correct outcome would be 1f3 2 per C's rules. Even changing the order of the read_bits_int() calls does not fix that.

[GreyCat]

What does it have to do with this code at all? If you're willing to compare it to C, please at least cast a byte array:

struct T {
  int16_t a : 12;
  int16_t b : 4;
};
char i[] = { 0x12, 0x34 };
struct T *v = (struct T *) i;

printf("%x\n%x\n", v->a, v->b); // output: 412 3

I get the exactly same output for read_bits_int_le.

[tempelmann]

But I wrote that the BE result is wrong. The LE is fine, just as you now again verified. The BE result should, in this last example, give 234 1 - but your read_bits code doesn't do that.

[GreyCat]

[GreyCat]

@JaapAap I believe your implementation is very close to the one I've just posted? Given that it's the way of packing bits that comes standard in all (?) C implementations, I wonder if we should support it as a default one, and return for more if and when the need for more complicated formats will arise?

Returning back to the very first example, parsing 3 bytes = 24 bits of data with 1+16+7 scheme, for a given x = [0x4a, 0x9d, 0xd0] would be something like that in C:

struct T {
  uint32_t a: 1;
  uint32_t b: 16;
  uint32_t c: 7;
};
char x[] = { 0x4a, 0x9d, 0xd0};
struct T *v = (struct T*) x;

printf("a=%x b=%x c=%x\n", v->a, v->b, v->c);

It will result in:

• On a big-endian machine: a=0 b=953b c=50
• On a little-endian machine: a=0 b=4ea5 c=68

For comparison, two methods from above yield b=ceca and b=3b95.