A really nasty bug
The Tight VNC Encoding has a number of different filters to improve data compression quality. One of those filters is "palette" which converts the pixel data into an indexed 8-bit representation. Before sending the data, the size of the palette (minus 1) is sent as a single byte and then the palette is sent followed by the filtered pixel data.
I had code that looked something like this:
uint8_t palette_size = ptr + 1; uint8_t *palette = ptr + 1; uint8_t *pixel_data = ptr + palette_size * bpp + 1; decode_length(pixel_data);
Strangely, when attempting to decode the length, I was getting an impossibly large number which resulted in garbage being passed to zlib resulting in a decompression error. So where's the error? It's subtle. If palette_size == 256, or ptr == 255, then palette_size silently overflows to 0 resulting in pixel_data actually being the palette. Curiously, I used a int for palette_size in the protocol parser so I wasn't having a problem there. Doh!