A really nasty bug

The Tight VNC Encoding has a number of different filters to improve data compression quality. One of those filters is "palette" which converts the pixel data into an indexed 8-bit representation. Before sending the data, the size of the palette (minus 1) is sent as a single byte and then the palette is sent followed by the filtered pixel data.

I had code that looked something like this:

uint8_t palette_size = ptr[0] + 1;
uint8_t *palette = ptr + 1;
uint8_t *pixel_data = ptr + palette_size * bpp + 1;

decode_length(pixel_data);

Strangely, when attempting to decode the length, I was getting an impossibly large number which resulted in garbage being passed to zlib resulting in a decompression error. So where's the error? It's subtle. If palette_size == 256, or ptr[0] == 255, then palette_size silently overflows to 0 resulting in pixel_data actually being the palette. Curiously, I used a int for palette_size in the protocol parser so I wasn't having a problem there. Doh!