Analysis of RealVNC Vulnerability

As many of you may have saw, Slashdot ran a story about a flaw found in RealVNC 4.1.1 that allowed an attacker to gain access to a password protected VNC server without any credentials.

By the time I saw this article, the site was already Slashdotted so I wasn't able to get a packet trace. The disclosure of the flaw was a bit troubling as there was no explanation of how it worked. Moreover, it sounded like it was stumbled upon by accident. My first thought was buffer overflow or integer overflow but it takes a very carefully crafted packet to exploit such flaws. Plus, VNC doesn't really have many variable length strings and certainly doesn't have any in the right parts of the protocol to explain this vulnerability.

Then it dawned on me while on campus the other day. Starting in version 3.7 of the RFB protocol, the security type negotiation was changed. It used to be that the server just sent back whatever security type it wanted to use. Common security types are rfbAuthNone (no password) and rfbAuthVnc (triple-des challenge/response). The new security type negotiation allowed the server to send back a list of security types letting the client choose which one it wanted to use.

Obviously, this makes supporting new security types more backwards compatible. A really silly mistake to make in the server would be to not check to make sure that the security type the client chooses is actually part of the original list the server supports. I looked at the RealVNC source code and sure enough, they weren't checking. What happened here is that the person who found this hole must have had a bug in his client where it sent the wrong security type (one that wasn't in the list).

This brings up a pretty interesting question. Why wasn't there a test case for this? This seems like a rather obvious thing to check for. Fortunately, this is really limited to RealVNC 4.1.x as all other VNC codebases branched off before the rewrite for 4.1.x and this problem was introduced in that rewrite. If you're using RealVNC, and you do not want password-less access to your machine, you should visit RealVNC and pick up the new version that fixes this problem.