PyGTK VncView Wrapper

It was amazingly easy once I fonud a developersWorks article on it. In a 34 line python app, I have a fully functioning VNC viewer that supports tabbing :-) Swank!

SetName VNC Encoding

I decided to hold off on the zlib reset VNC encoding extension. I'm contemplating whether recompressing is an option (I want to avoid protocol extensions at all cost).

However, one extension that I know is necessary is and not terribly contraversarily is updating the name of the desktop. This extension is very simple, a new pseudo-encoding that is sent as a U32 that is the size of the new name, followed by the name.

That's it.

Extending the RFB Protocol

I just realized that I needed to add a new extension to the RFB protocol to support my proxy. The problem is that each client maintains one or more zlib streams that are used to decrypt the data. These streams must be maintained used to process the data in order.

Proxying VNC sessions then can cause stream corruption because there's no way to signal the client to clear out the stream. How do we fix this?

Introduce a new encoding that the server issues to signal a switch to a new set of streams.

My old university happens to be in the most dangerous city in the US

According to Reuters, Camden, NJ, the home of the Rutgers campus I attended before transfering to UT, is the most dangerous city in the country for the second year in a row. The muder rate was 10 times the national average and the robbery rate was 7 times the national average.

And people asked why I transferred...

And from nowhere, SMP support in QEMU

Being on qemu-devel is like having a monthly Christmas present. Out of nowhere, Fabrice just checks in amazing things. A few weeks ago it was USB, and this evening, it was SMP support.

It's currently just guest SMP (the virtual CPUs are all multiplexed in the same process) but I wouldn't be surprised to see host SMP appear in the near future.

I wonder if SMP Xen will work under QEMU.

A really nasty bug

The Tight VNC Encoding has a number of different filters to improve data compression quality. One of those filters is "palette" which converts the pixel data into an indexed 8-bit representation. Before sending the data, the size of the palette (minus 1) is sent as a single byte and then the palette is sent followed by the filtered pixel data.

I had code that looked something like this:

uint8_t palette_size = ptr[0] + 1;
uint8_t *palette = ptr + 1;
uint8_t *pixel_data = ptr + palette_size * bpp + 1;

decode_length(pixel_data);

Strangely, when attempting to decode the length, I was getting an impossibly large number which resulted in garbage being passed to zlib resulting in a decompression error. So where's the error? It's subtle. If palette_size == 256, or ptr[0] == 255, then palette_size silently overflows to 0 resulting in pixel_data actually being the palette. Curiously, I used a int for palette_size in the protocol parser so I wasn't having a problem there. Doh!

VNC client library needs some work

The last real feature I needed for my vncviewer widget was Tight Encoding. As I mentioned in an earlier post, I had worked out a protocol parser so today I decided to sit down and write a decoder.

I ran into pixel format hell though as Tight uses packed RGB pixels as a special case when bits_per_pixel == 4. ZRLE uses it too but it's considerably easier to deal with in that case. This has led me to believe that it's time to factor out the decoding routines into a completely different library. Instead of providing drawing primatives, I think I really have to just give it a buffer like with QEmu.

I'm done for the night

My wrists are giving out.

I just can't help myself

I've been using gnome-blog-poster for my posts so far and while it's mostly nice, it uses the old XML-RPC interface for blogger which doesn't support titles. So, I've written my own that uses the new Atom based API.

Tight encoding and a new approach to proxying

I finished implementing support for Tight encoding in my VNC protocol library today. This is exciting because Tight is clearly the best encoding scheme that's supported by the largest number of servers out there.

I also had an interesting thought about how this should all work within Xen. I was envisioning a single daemon that was told to open a PTY and listen on two sockets (one for the incoming connections and one for the reverse connection).

I'm now thinking that a much more generic proxy is a better answer. The idea would be to have a VNC proxy that can be given an arbitrary number of VNC connections to multiplex (that are normal or reverse). A config file of some sort could define the key bindings used for switching. The key bindings would obviously have to be directional (alt-f7 to go from session 1->2 and ctrl-alt-f1 to go from session 2->1).

I would then just implement a VNC terminal program and run that as a normal VNC server.

An interesting idea

I just had a rather interesting idea. One downside of using Xvnc for a paravirtual framebuffer is that it requires a TCP connection to dom0. This is problematic for a few reasons. The first is that it assumes networking is up which is generally a big fat assumption. Imagine the use-case of a distro installation.

The second problem is that Xen supports network interface crediting which is going to be an important feature for a lot of users. A paravirtual framebuffer is going to use a lot of bandwidth so this is not practical.

While thinking about how much work it would be to port Xvnc to user a shared memory ring queue, I had a revelation. Why not create a daemon that listens on a socket in domain-U and then transfers all the data on that socket via a ring queue to then expose it (as another socket) in domain-0. You now have a dedicated shared memory transport that doesn't require networking. It should be relatively high performance (certainly better than just using a vif).

Much more elegant than something like dedicating a vif.

VNC Everywhere

I've been working with VNC a lot lately. You may or may not know that I rewrote QEMU's GUI to be a native GTK application (more info here). After seeing how easy this was and how much I enjoyed it, I decided I really wanted the same thing for Xen.

This means that the virtual framebuffer problem has to be solved. It's a big one that noone's wanted to touch so far and that we get an awful lot of criticism for.

A few of the Cambridge guys want to do the obvious. Implement a paravirtual framebuffer driver in Linux. They even have some smart ideas about strifing it across pages to identify update regions. I don't like this approach though because it's going to have a noticable performance impact (event if noone's connected, we still have to render the console text to the buffer) and it's a lot of work.

I have a different approach. The idea is to leverage Xvnc (as many do right now with Xen). In domain-0, we would have a fake VNC server that connects a domain-U's emergency console and renders the console to VNC. This is essentially a VNCTerm program and not an amazingly new concept. The clever part in this story though is having this fake VNC server also play the role of a VNC proxy by setting up the domain-U's X server to actually be Xvnc configured to use reverse VNC to connect back to the fake VNC server. When X starts up in the domain, the fake VNC server would accept the reverse connection and do a virtual VT switch and begin acting just a VNC proxy.

By utilizing the DesktopSize VNC psuedo-encoding, this would give the appearance of Just Working. The fake VNC server can also trap ctrl-alt-fN key events to switch back to the emergency console.

If the distro's are willing to play ball here, this can be easily extended for graphical installers too since SUSE has had support for installations over VNC for quite some time and RedHat would simply have to launch Xvnc to run Anaconda in.

I'm somewhere between the proof-of-concept and alpha stage right now. I've written my own VNC protocol parsing library (that's a whole other post), a GTK VNC viewing widget, a VNCTerm server, and a VNC proxy (using the protocol library). Everything needs a lot of cleanup and some bug fixes. The performance is already very good so I'm quite happy about that. With any luck, Xen 3.1 will be extremely easy to use.

A little introspective

I've maintained many blogs in my time but they all have been self-hosted and using software that I wrote. This meant blogging involved quite a bit of work on my part.

I've been wanting to start blogging about Xen development for some time now but have not wanted to invest the time in blogging again so I thought I'd give using blogger a go.

We'll see how I like it. For now, please use http://blog.codemonkey.ws to access this blog and you'll always go to the right place.

first post

first post!