Tales of a Code Monkey

Red Hat announces "next-generation" virtualization based on KVM

2008-06-18T14:21:00.000-07:00

Today, at the Red Hat Summit, Red Hat announced three virtualization initiatives including oVirt. The press release is here.

Some choice quotage:

KVM technology has rapidly emerged as the next-generation virtualization technology, following on from the highly successful Xen implementation.

Another good one:

We continue to see huge improvements in functionality, performance and time to market because of our close relationship with our open source partners. For example, Intel and IBM have worked with us for many years covering virtualization technologies that span from Red Hat Enterprise Linux 5 to today's KVM-based announcements.

And of course:

"IBM works closely with Red Hat and the open source community to drive innovation within the Linux kernel," said Daniel Frye, vice president, open systems development at IBM. "IBM has a heterogenous approach toward virtualization, with KVM one of several options. KVM leverages the core features of the Linux kernel, including paravirtualization interfaces contributed by IBM engineers. By combining Linux virtualization infrastructure with open management interfaces such as CIM and libvirt, we gain a solution that eliminates lock-in and open source community innovations, we are able to offer our customers a solution with outstanding performance, scalability and agility."

If you want to see what all the fuss is about, check out KVM.

KVM and Green Computing

2008-06-09T12:20:00.000-07:00

A ran across this article today from Tom Henderson that draws attention to the fact that most existing hypervisors (ESX, Xen, Hyper-V) do not support frequency scaling and therefore are not very eco-friendly.

This is partly true. There has been some recent work in Xen to add deep sleep state support and I believe even some work on frequency scaling. It is certainly not true though that virtualization and power-consciousness are at odds with each other. KVM is able to leverage all of the work that's been invested into Linux to manage power wisely. Good power management does not cause any sort of performance drop. Reducing the performance of your workload is only going to make the machine run longer and consume more power.

The reason most hypervisors don't support power management is that it's very hard. When inventing a new Operating System, there's a lot of things you have to focus on before you can even start looking at power management. Again, we see the benefits of using an existing Operating System for virtualization.

The truth about KVM and Xen

2008-05-09T15:16:00.001-07:00

When I saw this article in my inbox, I knew I shouldn't bother reading it. I really couldn't help myself though. I'm weak for gossip and my flight was delayed so boredom got the best of me.

I can't blame the tech media for the wild reporting though. The situation surrounding KVM, Xen, and Linux virtualization is pretty confused right now. I'll attempt to do my best to clear things up. I'll make an extra disclaimer though that this is purely my own opinions and does not represent any official position of my employer.

I'm think we can finally admit that we, the Linux community, made a very big mistake with Xen. Xen should have never been included in a Linux distribution. There, I've said it. We've all been thinking it, have whispered it in closed rooms, and have done our bests to avoid it.

I say this, not because Xen isn't useful technology and certainly not because people shouldn't use it. Xen is a very useful project and can really make a huge impact in an enterprise environment. Quite simply, Xen is not, and will never be, a part of Linux. Therefore, including it in a Linux distribution has only led to massive user confusion about the relationship between Linux and Xen.

Xen is a hypervisor that is based on the Nemesis microkernel. Linux distributions ship Xen today and by default install a Linux guest (known as domain-0) and do their best to hide the fact that Xen is not a part of Linux. They've done a good job, most users won't even notice that they are running an entirely different Operating System. The whole situation is somewhat absurd though. It's like if the distributions shipped a NetBSD kernel automatically and switched to using it when you wanted to run a LAMP stack. We don't ship a plethora of purpose-built kernels in a distribution. We ship one kernel and make sure that it works well for all users. That's what makes a Linux distribution Linux. When you take away the Linux kernel, it's not Linux any more.

There is no shortage of purpose-built kernels out there. NetBSD is a purpose-built kernel for networking workloads. QNX is a purpose-built kernel for embedded environments. VxWorks is a purpose-built kernel for real-time environments. Being purpose-built doesn't imply superiority and Linux currently is very competitive in all of these areas.

When the distros first shipped Xen, it was done mostly out of desperation. Virtualization was, and still is, the "hot" thing. Linux did not provide any native hypervisor capability. Most Linux developers didn't even really know that much about virtualization. Xen was a pretty easy to use purpose-built kernel that had a pretty good community. So we made the hasty decision to ship Xen instead of investing in making Linux a proper hypervisor.

This decision has come back to haunt us now in the form of massive confusion. When people talk about Xen not being merged into Linux, I don't think they realize that Xen will *never* be merged into Linux. Xen will always be a separate, purpose-built kernel. There are patches to Linux that enable it to run well as a guest under Xen. These patches are likely to be merged in the future, but Xen will never been a part of the Linux kernel.

As a Linux developer, it's hard for me to be that interested in Xen--for the same reasons I have no interest in NetBSD, QNX, or VxWorks. The same is true for the vast majority of Linux developers. When you think about it, it is really quite silly. We advocate Linux for everything from embedded systems, to systems requiring real-time performances, to high-end mainframes. I trust Linux to run on my dvd player, my laptop, and to run on the servers that manage my 401k. Is virtualization so much harder than every other problem in the industry that Linux is somehow incompatible of doing it well on its own? Of course not. Virtualization is actually quite simple compared to things like real-time.

This does not mean that Xen is dead or that we should have never encouraged people to use it in the first place. At the time, it was the best solution available. At this moment in time, it's still unclear whether Linux as hypervisor is better than Xen in every scenario. I won't say that all users should switch en-masse from Xen to Linux for their virtualization needs. All of the projects I've referenced here are viable projects that have large user bases.

I'm a Linux developer though, and just as others Linux hackers who are trying to make Linux run well in everything from mainframes to dvd players, I will continue to work to make Linux work well as a hypervisor. The Linux community will work toward making Linux the best hypervisor out there. The Linux distros will stop shipping a purpose-built kernel for virtualization and instead rely on Linux for it.

Looking at the rest of the industry, I'm surprised that other kernels haven't gone in the direction of Linux in terms of adding hypervisor support directly to the kernel.

Why is Windows not good enough to act a hypervisor such that Microsoft had to write a new kernel from scratch (Hyper-V)?

Why is Solaris not good enough to act as a hypervisor requiring Sun to ship Xen in xVM? Solaris is good enough to run enterprise workloads but not good enough to run a Windows VM? Really? Maybe :-)

Forget about all of the "true hypervisor" FUD you may read. The real question to ask yourself is what is so wrong with these other kernels that they aren't capable of running virtual machines well and instead have to rely on a relatively young and untested microkernel to do their heavy lifting?

Update: modified some of the text for clarity. Flight delayed more so another round of editing :-)

KVM Forum 2008 Call For Presentations

2008-04-07T21:22:00.000-07:00

This is the Call for Presentations for the second annual KVM Developer's Forum, to be held on June 10-13, 2008, in Napa, California, USA [1]. We are looking for presentations on KVM development, quality assurance, management, security, interoperability, architecture support, and interesting use cases. Presentations are 50 minutes in length; there are also 25-minute mini-presentation slots available.

KVM Forum presentations are an excellent way to inform the KVM development community about your work, and to gather valuable feedback about your approach.

Please send your presentation proposal to the KVM Forum 2008 Content Committee at kf2008-cfp@qumranet.com by April 20th.

KVM Forum 2008 Content Committee:
Dor Laor
Anthony Liguori
Avi Kivity
[1] http://kforum.qumranet.com/KVMForum/about_kvmforum.php

On a personal note, I found KVM Forum 2007 to be one of the best run conferences I've attended. The facilities were great and each talk was interesting. There was a great deal of discussion during each talk. Definitely worth the trip.

KVM for the Mainframe

2008-04-06T16:31:00.000-07:00

kvm-65 was released today. The most interesting feature in this release is support for the s390 architecture, more specifically, the System z9 line of mainframes.

The s390 is the grand-daddy of virtualization. Everything started there. In so many ways, everything we're doing with x86 virtualization is just playing catch-up. The new exciting features like hardware virtualization support and hardware paging support have been in s390 forever.

s390 clearly has a very mature hypervisor. What many people may not know though is that it's normal to run two hypervisors at any given time on s390. At the bottom level, there's PR/SM which divides the machine into rather coarse partitions. Within a PR/SM partition, you can run z/OS or Linux. You can also run z/VM within a PR/SM partition. z/VM is another hypervisor that allows for much more sophisticated features like memory overcommit and processor overcommit. The user has the ability to decide how much hypervisor they need to maximize the efficiency of their workloads.

Within a z/VM partition, you can run z/OS or Linux. The beauty of s390 is that this configuration has been supported in the hardware for many years and is very fast.

When Linux adopted native support for virtualization, it became obvious that this could be easily supported on the s390. The hardware has long supported this sort of nested virtualization and the implementation turned out to be very straight forward. It helps that the x86 virtualization extensions were inspired by a paper written about s370 almost 30 years ago :-)

What do you get from a platform that has supported virtualization for longer than I've been alive? In this very first release of KVM for s390, it already supports 64-way guests. After two years of development, we've just gotten to supporting 16-way guests on x86.

Exploiting live migration

2008-03-19T08:40:00.001-07:00

Apparently at this year's BlackHat, someone presented a paper about attacking live migration traffic. The paper describes a tool called Xensploit which uses a man-in-the-middle attack on live migration traffic to do all sorts of bad things. The core problem is that Xen live migration is not encrypted. Neither is VMotion traffic so the exploits are equally applicable.

While there's already been a lot of commentary suggesting that live migration shouldn't happen over insecure networks, that's not good enough for me. If you are sending the memory of a VM over the network unencrypted, you might as well not have any passwords on any of your machines since you are exposing all of the VM's sensitive data to anyone on the network.

For IBM Director Virtualization Manager, we go to great lengths to always ensure that Xen live migration traffic is always encrypted. As far as I know, no other Xen management tool is capable of encrypting live migration traffic. If you are using Virtualization Manager, you are protected from Xensploit style attacks.

For KVM, we were careful not to make the same mistakes that had been made for Xen. KVM supports live migration over SSH by default and provides a mechanism for third-parties to encrypt migration traffic in anyway they please.

A preview of gtk-vnc v0.3.3

2008-01-12T10:16:00.000-08:00

Since Dan beat me to blogging about the gtk-vnc 0.3.2 release, I decided to co-opt him for 0.3.3 and post a full two weeks before the release actually happens :-)

The 0.3.3 release will add support for the Tight encoding which is perhaps the most widely supported compressed encoding out there. This was really the last piece in making gtk-vnc a first class VNC client supporting all the protocol options that one would expect a good client to support. Much to my surprise, 0.3.3 will also contain a Firefox plugin that allows a VNC widget to be embedded within your web browser thanks to Rich Jones.

At first, a VNC web-browser plugin may sound like a silly idea. Of course, both RealVNC and TightVNC ship a Java applet VNC client. Clearly, there is demand for embedding a VNC session within a web browser. Besides the obvious concerns about performance, Java applets are severely limited in what they can do. You cannot grab the mouse and you cannot grab arbitrary key events. You really can't build a first class VNC client as a Java applet.

With a gtk-vnc based plugin, you can have a first class VNC client in your web browser. An exciting application of such a technology would be a rich web-based management application for virtualization. Things that were not possible in Java, like full-screening a VNC session, supporting copy/paste and drag-n-drop, are all within the realm of possibility using a gtk-vnc plugin.

There's still a fair bit of work to do to harden the plugin and gtk-vnc, such that it could be trusted to be invoked by any web page, but I'm looking forward to see what this leads to.

First release of extboot

2007-12-05T14:23:00.000-08:00

Today I released the first set of patches for extboot. extboot is an option ROM that allows booting a guest from virtually any type of block device.

Historically, the PC BIOS is only capable of booting from IDE devices. The PC BIOS doesn't need a special driver for every type of IDE controller simply because every IDE controller supports a compatibility mode that dates back to the earliest IBM PCs. The PC BIOS uses this compatibility mode to access the disk thus avoiding having to support dozens of different IDE controllers. When SCSI was introduced, to allow these devices to used for boot, option ROM support was added to the PC BIOS. Every PCI device can provide a piece of ROM memory that the BIOS runs before booting. These ROMs can do horrible things to overwrite portions of the BIOS and trick the BIOS and bootloaders into thinking they are booting from an IDE device when it's really booting from a SCSI device.

Most virtualization solutions offer support for IDE and SCSI devices and include SCSI option ROMs to enable booting from SCSI. Some products, like Xen, also provide paravirtual disk drivers. Up until now, these devices were not bootable. This required guests to have a bootable IDE partition and then another PV disk partition. It's a real pain from an administration perspective. Beyond performance, there are a few reasons to prefer PV disk drivers over SCSI. PV disk drivers allow unlimited support for adding new features whereas with SCSI you are limited to whatever hardware supports.

extboot is an option ROM that can trick the PC BIOS into thinking that any block device is actually an IDE drive. It can be used not only for booting from SCSI devices but also from true PV disk drivers. This is something that, to the best of my knowledge, has never been possible in any x86 virtualization solution.

extboot support should be available in QEMU, KVM, and Xen in the near future so keep an eye out for it :-)

CIM support for KVM and Xen

2007-11-05T08:39:00.001-08:00

As announced on the libvirt list today:

This is the announcement of a new open-source project called libvirt-cim based on libvirt and aiming at offering the complete functionality of libvirt via a CIM provider implementing the DMTF SVPC virtualization model and released under an LGPL licence. A CIM [1] provider is an implementation of a set of standardized interfaces (Common Information Model) whose goal are to provide well defined entry points allowing easier and interoperable management tools to be built. In the case of libvirt-cim, the goal is of course to export the SVPC virtualization model, which then can be used to manage storage, hosts and domains remotely.

Since this new CIM provider is based on libvirt, it supports QEMU, KVM, Xen, and potentially much more. The provider is already quite functional and was developed at IBM by Dan Smith, Jay Gagnon, and Heidi Eckhart. See the announcement for more information. Update: For clarification, the CIM provider only supports Xen today but it should very easy to add support for the other VMMs supported by libvirt.

TPR patching

2007-10-29T15:55:00.000-07:00

I'm heading off to Japan tomorrow morning for the Linux Foundation Japan Symposium but instead of packing like I should, I figured I'd post about an exciting new feature in KVM.

First, a little background. Even when doing hardware accelerated virtualization (using VT or SVM), there is a lot of emulation that is required for IO devices. While there are probably at least 15-20 different devices that must be emulated for a virtual machine, there are only a few that are performance sensitive. The two most notable are the network card and disk controller. Since all Operating Systems support a wide variety of these devices, we can create a fake network card driver that we can emulate in a high performance way and everything works out nicely (these are commonly called paravirtual device drivers).

There are some devices in the modern PC that you cannot write drivers for because there simply aren't that many of them. For instance, there are really only a couple kinds of interrupt controllers so most Operating Systems don't provide a mechanism for loading interrupt controller device drivers. Instead, these devices are baked in deeply within the Operating System's core.

For the most part, none of these devices affect performance significantly. The notably exception is the local APIC. The local APIC is a per-processor interrupt controller whose interface is memory-mapped. This means that an OS communicates with the local APIC by writing to a special memory location. In particular, the local APIC has a feature called the TPR (task priority register). Certain OS's (namely, Windows), access the TPR extremely frequently. If you've used Windows under KVM, you may be familiar with the ACPI work-around which effectively tricks Windows into thinking there isn't a local APIC. The result is a significant increase in performance since we no longer have to emulate thousands of TPR accesses per-second. Unfortunately, ACPI is a useful thing. You can't have SMP without it. Disabling it is not really a great solution to the problem.

At this past KVM Forum, Ben Serebin , from AMD, shared an interesting observation. Windows guests only access the TPR with instructions that are at least 5 bytes. The significance of 5 bytes is that that happens to be the size of an absolute call on the x86. This means that you can replace any of the TPR access instructions with an absolute call without the need to do fancy dynamic translation. If you're very clever about hiding routines within the BIOS (it turns out, Windows always has a valid virtual mapping to the BIOS), you can actually rewrite TPR access instruction to instead be calls to functions, that you provide, that access the TPR in a more efficient way.

Avi Kivity posted an implementation of this to KVM recently. The results are quite dramatic. Windows XP installs are at least twice as fast--perhaps even faster. The very latest Intel processors have a hardware feature that ends up with the same result but the nice thing about a purely software approach is that it will work with older processors.

This code hasn't made it's way into a KVM release yet as it needs a bit more testing and clean-up. I suspect we won't see it in a release for a couple more weeks but once it's there, you can reenable ACPI in your Windows guests and enjoy good performance :-)

The Myth of Type I and Type II Hypervisors

2007-10-08T13:42:00.001-07:00

This has been something that has bothered me for a while that I have never gotten a chance to articulate. In the virtualization community, the terms "type-1" and "type-2" hypervisors get thrown around a lot--often carrying different meanings. Lately, "type-2" is being used as a derogatory term suggesting that a virtualization solution is "lesser" than a true "type-1" hypervisor.

The most common definition of "type-1" and "type-2" seem to be that "type-1" hypervisors do not require a host Operating System. In actuality, all hypervisors require an Operating System of some sort. Usually, "type-1" is used for hypervisors that have a micro-kernel based Operating System (like Xen and VMware ESX). In this case, a macro-kernel Operating System is still required for the control partition (Linux for both Xen and ESX).

The whole argument of micro-kernel vs macro-kernel hosts is a different blog post (just as a spoiler, I think one can make a better argument for macro-kernel hypervisors). I want to focus, instead, on why we have these terms and what they really mean.

Virtualization theory really started with a paper from Gerald Popek and Robert Goldberg called Formal Requirements for Virtualizable Third Generation Architectures. The paper is a mathematical proof of the architectural requirements to allow virtualization. It is very terse and I don't expect most people have read it. The paper focuses on implementing full virtualization on native hardware and focuses on things like whether privileged instructions are trappable. It was written in 1974 and Operating Systems were not actually all that common back then. Many people think the terms "type-1" and "type-2" originated from this paper but that is simply not the case. The paper does mention the concept of recursive virtualization and briefly discusses the requirements to allow one virtual machine to run within another virtual machine.

As best as I can tell, the terms "type-1" and "type-2" originate from a paper by John Robin called Analyzing the Intel Pentium's Capability to Support a Secure Virtual Machine Monitor. This paper was Robin's master thesis at the Naval Postgrade School. There are two versions of the paper available, the actual master's thesis and a condensed version for USENIX 2000.

This paper is really an application of the Popek/Goldberg proof to the Pentium architecture. A few points were missed, but it does a rather good analysis of why the Pentium architecture did not satisfy the Popek/Goldberg requirements for virtualization. Now, some folks at VMware have made a rather compelling case that this is in fact incorrect because the Popek/Goldberg proof does not eliminate the possibility of using dynamic translation. At any rate, Robin makes a distinction between "type-1" and "type-2" VMMs. The reason for the distinction is simple. When discussing "type-1" VMMs that access hardware directly, the set of requirements to enable Secure Virtualization entirely depends on the hardware. When discussing "type-2" VMMs, however, you do not have direct access to hardware so the requirements to enable virtualization are actually at the Operating System interface. A true "type-2" VMM is just a process in an Operating System and is not capable of accessing hardware directly.

The important point to take away here is that all modern virtualization solutions (except for unaccelerated QEMU maybe) are technically "type-1" VMMs according to Robin. The things commonly cited as "type-2" VMMs like VMware Workstation, Parallels, VirtualPC, and KVM all rely on kernel modules which means they do have direct access to hardware. This makes all of these solutions "type-1" VMMs. What's more important though is that the distinction of "type-1" and "type-2" has absolutely no bearings on performance, robustness, or any other qualitative factor. It is merely a distinction made when attempting to formulate a proof about whether virtualization is possible or not. It starts to lose meaning too when an Operating System is capable of supporting a true "type-2" VMM (which arguable, the KVM interface in Linux enables). Does that mean that Linux is a "type-1" VMM and QEMU using the KVM interface is a "type-2" VMM? How can the same solution be both though? IMHO, the introduction of the term "type-2" was really a mistake on Robin's part perhaps as a misunderstanding of the section of the Popek paper regarding recursive virtualization. That's just speculating though. The distinction really just doesn't make much sense in my mind.

So if you've made it this far, I'll hope you agree that these terms really have no practical meaning and will join me in refraining from using them in the future :-)

Coherence for QEMU

2007-02-28T23:20:00.000-08:00

As I have previously discussed, I have been fascinated by the idea of Coherence that Parallels has now officially supporting. After some digging, I think I have a pretty good idea of how it works.

Very similar technologies exist. SeamlessRDP is a special program you can run in a Terminal Services session to expose only a single application over RDP. It works by replacing the normal Shell program (explorer.exe) with a process that uses SetWindowsHookEx to keep track of window creation, destruction, resizing, and movement events. For SeamlessRDP, this information is sent over a special RDP channel.

The RDP session is always full screen and this window position information is used to only show the portion of the RDP session that the window occupies. Since the RDP session is full screen, and the window positions are mapped at the same location in the host as in the RDP session, things like z-order and window dragging Just Work.

To just get the taskbar, you just have to launch explorer and track it's children being careful to not display the desktop window. Here is a screenshot demonstrating this with QEMU, KQEMU, SeamlessRDP, and a slightly modified rdesktop.

I'd like to integrate this all a little more into QEMU. The first thing I'd like to do is eliminate the need for RDP. We can use a paravirtual channel to communicate the windowing information and then just use VNC extensions to communicate that data to the client. A tricky problem is that the session has to be full screen for this to work and QEMU does not provide VGA emulation that supports some weird resolutions (1400x1050--which my laptop uses!). I think this can be solved with software scaling though.

KQEMU is now free software!

2007-02-06T07:30:00.000-08:00

As part of the 0.9.0 release, Fabrice Bellard released KQEMU under the GPL. KQEMU is an accelerator for QEMU that works on older hardware (without hardware virtualization). As part of this release, Fabrice also published detailed technical notes.

You can get the GPLv2 version of KQEMU here. I want to thank Fabrice for doing this. There are a lot of people in the QEMU community who are very happy about this.

QEMU 0.9.0 is now available

2007-02-06T07:28:00.000-08:00

This release has been in the works for quite a while. A whole bunch of changes went in. The official changelog is:


version 0.9.0:

  - Support for relative paths in backing files for disk images
  - Async file I/O API
  - New qcow2 disk image format
  - Support of multiple VM snapshots
  - Linux: specific host CDROM and floppy support
  - SMM support
  - Moved PCI init, MP table init and ACPI table init to Bochs BIOS
  - Support for MIPS32 Release 2 instruction set (Thiemo Seufer)
  - MIPS Malta system emulation (Aurelien Jarno, Stefan Weil)
  - Darwin userspace emulation (Pierre d'Herbemont)
  - m68k user support (Paul Brook)
  - several x86 and x86_64 emulation fixes
  - Mouse relative offset VNC extension (Anthony Liguori)
  - PXE boot support (Anthony Liguori)
  - '-daemonize' option (Anthony Liguori)

But this is just scratching the surface. You can obtain it from the usual place.

KVM, Xen, and the Linux kernel

2007-02-06T07:15:00.000-08:00

I stumbled upon an article from DevX where Ian Pratt is quoted on a number of topics including KVM and upstream merge. I thought what he said about KVM was a little odd, but what disturbed me was that I think the interviewer misinterpreted what Ian said re: upstream merge. Ian said:

Putting Xen into Linux doesn't make sense: hypervisors are different beasts from operating systems, so they share little code.

He's referring to putting the actual hypervisor into the kernel. Unfortunately, the interviewer took this to mean:

Pratt also explained that Xen is no longer actively seeking inclusion in the mainline Linux kernel either.

Which is totally missing the point. We've never wanted the hypervisor to be included in mainline Linux. It's not a part of Linux so I don't even see how we would do it without major rewrites. What we've been trying to get into the kernel is the Linux changes for guest that run on top of the hypervisor.

We are still very interested in getting the Linux changes upstream. In fact, this is a major priority.

Migration in QEMU

2007-01-17T11:54:00.000-08:00

For a long time, I've thought about QEMU migration. For KVM, Qumranet added a static form of migration to QEMU. I've been working with Xen migration for a while now (mostly in the scope of IBM products) and I certainly have learned a lot from it. Honestly, I don't really like how KVM is doing migration so after spending a weekend heads down on V2E, I decided to take some time and implement migration for QEMU.

The biggest problem I have with KVM and Xen's migration is that it uses open TCP ports. This is just such a bad idea. It's a security nightmare to transfer the contents of memory over an unencrypted channel. For QEMU, I decided to allow the user to spawn an external program to setup the channel to send the migration traffic over. This lets a user just use SSH or RSH if they want something that works. This also let's management tools implement their own mechanism. This may use OpenSSL, CIM, or any other mechanisms out there. It also provides a mechanism for implementing some interesting things like light weight checkpointing (although that's another topic).

This does make things a bit more complicated though. Instead of just saying 'migrate hostname' you now have to construct a rather long command like 'migrate "ssh hostname qemu -loadvm -"'. A nice side effect though is that you can completely change the command line arguments in case you have NFS mounts at different locations.

At the moment, I have a static migration patch. I'd like to implement live migration real soon. I think it will be pretty easy. It's just a matter of adding a new set of callbacks to allow devices that may take a long time to save/restore to instead, provide a "live" save/restore callback. We'll just run through the live callbacks first and when they've signaled that they're done, we'll go ahead and activate the non-live callbacks. This probably will only touch the RAM save/restore code at first which seems more than okay to me.

If you're interested in taking a peak, you can just take a look in my QEMU patch queue.

Binary kernel modules are dead in 2008--now what

2006-12-13T19:45:00.000-08:00

It was decided today on LKML that starting in January 2008, binary modules are no longer going to be loadable in the Linux kernel. This has some rather major consequences for a number of virtualization technologies.

Parallels, Win4Lin, kqemu, and VMware rely on binary modules for their Linux products. I suspect all of these products will have a hard time moving their code out of kernel space seeing that it's performance sensitive. So what are they all going to do? I see three possible options: 1) drop Linux support (Win4lin and kqemu disappear completely) 2) build a minimal kernel interface to privilege state and try to develop fast userspace interfaces. I can't see how one could do a fast userspace shadow paging implementation though. 3) open source the kernel bits.

Everyone's focused on management now right? Can you imagine if the VMware binary translator was GPL'd? Kudos to the kernel developers for finally doing the right thing here.

Update: Linus is insisting that the distros merge this patch first before he'll take it.

Coherence for the rest of us?

2006-12-06T20:20:00.000-08:00

There's quite a buzz about Parallels new coherence technology. In principle, there's nothing that exciting going on here from a technology perspective but what we have is the result of competition. Basiclally, Parallels is serving a very useful market that VMware has forgotten -- the minority home user. Coherence allows you to run a virtual machine and have individual applications display their windows in the host OS. Essentially, you have a small program running in the guest OS that exposes the window for an app in the host. This is similar to what Meta-VNC is already doing with VNC. The nice thing is that they've packaged it all up in an easy to use form. So, running Windows apps on Mac OS X is nice and all but what about us Linux users? Well, I've done a little bit of research here and it looks like there are a number of tricks you can do in Windows to get an image of an app. The best way seems to be WM_PRINT/WM_PRINTCLIENT except that it requires support from the application. I don't know how many apps support this since WM_PRINTCLIENT is not handled by the default message handler. I'm thinking of starting though with the opposite case. Let's expose a single application from a Linux guest via VNC. The obvious way to do this is with a special X server. You simply launch your VNC X server on a new display, and then launch your app with the proper DISPLAY environmental variable set. We'll need a custom X server that can actually know how to render the individual windows of course. Plus, popups are going to require some VNC extensions. It should end up being pretty neat though. Definitely a fun little project. Is it useful to run Linux apps under Windows? I don't know. It's a fun project though :-)

Novell Sells Out (In the name of virtualization)?

2006-11-02T20:22:00.000-08:00

Groklaw says it best. Maybe the GPLv3 isn't so bad after all...

Common Neutral Hypervisor? Trademarks and the GPL

2006-10-07T19:35:00.000-07:00

I love Free Software. Like many free software developers, I'm rather concerned about the recent debates regarding the role of trademarks and free software. Just five years ago, the idea that trademarks would be a problem with free software was almost laughable. With high profile projects like Firefox developing questionable trademarking policies, the question of how trademarks affect free software is becoming very important. As I write this, there's a heated debate within the Xen community over XenSource's new trademark terms specifically regarding the Xen trademark. Suffice to say, that the terms concern RedHat enough that they've announced that they're considering renaming Xen to CNH, or Common Neutral Hypervisor. They appear to be concerned that they can not live up to the trademarking terms. Personally, I'm not in a position to comment about the new trademark policy. I try to keep my nose clean of this sort of thing. However, it's times like this that I realize how important the GPL is in defining what free software is and how lost we would be without it (or at least, how much arguing there would be). I wonder if there's enough room in the GPLv3 process to introduce trademark terms...

Optimizing VNC for localhost

2006-10-03T19:01:00.000-07:00

I've got some free time now and have been thinking recently about revisiting QEMU GUI support. Previously, I had a set of patches that implemented a shared memory transport for QEMU's graphic interface. The first change I wanted to make to my old patches, was to use a TCP transport instead of QEMU's char device interface. I quickly realized though that there would be a lot of shared code between this new transport and the VNC transport. At this point, I started thinking about what it would take to add a shared memory transport to VNC. Conceptionally, all this would require is a new encoding type that can send back a shared memory ID. The client would have to send a little more than just a SetPixelFormat though since the bytes-per-line is also needed. What this would allow though, is for the server to allocate a shared memory segment, hand that info over to the client, and the client could then hand that over to the X server. This would have fantastic performance on the localhost case. Reusing the VNC protocol means a much simpler client. I sent off a note to the VNC folks asking to reserve a pseudo-encoding range. Once I get a response, I should be able to hack something up fairly soon.

Finally back to normal

2006-09-14T15:26:00.000-07:00

While it took a bit longer than it should have, I've finally gotten my IBM accounts straightened out again. I've decided to ditch one of them but chances are if you've ever sent me mail at my IBM address, you were using that one. If you gotten a vacation message from my IBM email address or a bounce, you shouldn't have a problem anymore.

Back in Austin

2006-09-11T07:06:00.000-07:00

I'm now back in Austin after quite a lot of travel. I'm officially back at IBM now and should resume blogger as before about my work on Open Source virtualization. My email should now be back to normal. I will be going through the unanswered mail in my INBOX over the next few days. I've had a lot of email trouble lately (which has all now been resolved) so if you don't get a response to something you think I should have responded to, I may have not seen it (or I'm just being lazy :-)).

More email nonsense

2006-08-22T05:22:00.000-07:00

Someone gave me a heads up that aliguori at us ibm com is bouncing right now. I took some time off before switching to being a full timer at IBM (to spend some time w/family and do a little traveling). I go back to IBM on September 6th. I was expecting that my ibm accounts would stick around until then but it doesn't seem that way... Either way, my codemonkey.ws or utexas accounts can be used instead.

Email Glitch

2006-08-21T17:14:00.000-07:00

My cable modem gave out late last night and since I'm out of town I won't be able to restart the stupid thing until September. Fortunately, I was able to bring up a quick relay so that @codemonkey.ws email is now going somewhere. I don't think it was down long enough for retries to timeout but if you sent me an email between midnight-8pm CST today, you may have to resend. If mail servers are caching the MX record aggressively, it'll definitely bounce. On a positive note, I discovered that I can use my utexas account as an authenticated SMTP relay for @codemonkey.ws. Lately, I've been getting a lot of outgoing mail rejected b/c my cable modem's network is on a few blacklists since they're dynamic IPs. This should provide a solution until I get back to Austin and colocate a real server somewhere.

Spend more time reading, less time watching "The Matrix"

2006-07-06T17:23:00.000-07:00

I was recently pointed to two different programs that attempt to detect the presence of a VMM and produce an undetectable VMM. These are called "red pill" and "blue pill" respectively and both are by Joanna Rutkowska of invisiblethings.org. "red pill" claims to detect modern VMM's by checking the results of a sidt. Since most VMM's hide themselves in upper memory, the claim is that if the IDT (which is of course being shadowed) is in the upper 256MB of memory, you're in a VMM. Most operating systems tend to use the upper portion of memory to map physical pages so this is probably going to work more often than not. However, it's quite easily defeatable since in the VMM you could either 1) move yourself to a lower bit of memory or 2) just use a full emulator and avoid any shadowing. Therefore, the "red pill" is really not that useful. This doesn't mean that you can't detect a VMM's presence. If you go all the way back to the Popek/Goldberg paper (which is 30 years old), they make it quite clear that any program that depends on timing will no longer function as expected in a virtual machine. One can exploit this fact to detect the presence of a VMM in a generic way. This is exactly what is done in Pioneer. Pioneer makes use of a checksumming algorithm that also incorporates EFLAGS. Since pushf is not trappable and EFLAGS is really hard to rewrite with a JIT (since you have to keep track of condition flags), one can easily detect (by checking how long the checksum takes to compute) the presence of a VMM. So what about "blue pill". The author seems to think that SVM is impossible to detect because the VMM no longer has to shadow things like the IDT. Of course, this is naive since the one can still do a timing analysis. Okay, so I'm being pretty harsh here. This is all very obscure stuff. I'm somewhat amazed though because I've seen references to these things in a number of places now (including major news sites).

Ajax terminals

2006-05-26T21:59:00.000-07:00

I recently ran across AjaxTerm. It's pretty neat. It got me rather excited as I think it would be a really cool part of a xen Web-based management console.

I was rather curious how it worked with respect to updates. Clearly, when you send console data it requires an HTTP request. That's unavoidable. However, how do you detect when it's time to receive console data? There's no such thing as a reverse HTTP connection so that's out of the question. A naive solution would be to poll the server. That seems rather wasteful though. More importantly, there is an inevitable lag depending on how tightly you loop.

That got me thinking about an alternative approach this evening. What if you started a request (asynchronous of course) and the server simply didn't respond right away. Even if the browser eventually times out the connection, as long as it lasts for a pretty good portion of time, you could use this as a way to wait (without polling) for new data to arrive. So far, some experiments have shown promise. We'll see whether this results in a noticeable effect on latency.

Turning on optimization gives better warnings?

2006-05-24T07:16:00.000-07:00

This could be one of those things that everyone knows except for me, but I learned a very hard lesson today. We had a code review internally for a bit of code I've been doing. A very embarrassing bug turned up that was the result of a poorly tested last minute query-replace.

The crazy thing was that it *should* have been caught by the compiler as an uninitialized variable. I had -Wall in my CFLAGS. Sure enough though, no matter how blatant I made the uninitialized usage, the compiler said nothing.

I looked in the info pages and discovered that uninitialized values are only reported if -O is present. -O does the necessary bookkeeping to allow data flow analysis which is clearly required for reporting of uninitialized values. I typically don't enable -O but I guess most projects I work on do. I will certainly be using -O from now on though.

Some interesting stats on sensitive instruction usage

2006-05-19T09:41:00.000-07:00

One surprising thing about x86 virtualization is that while there is a small number of instructions that require special effort (around 17 depending on what you count), there are an even smaller number that make 95% of all the calls to these instructions.

These four instructions are cli, sti, pushf, popf (which probably isn't all that surprising). An interesting result of this is that you only really have to focus on making these instructions fast. This is an important conclusion when discussing the performance trade-offs of supporting a transparently paravirtualizable kernel.

Rusty Russell posted some stats that confirm this here.

Xen Becoming a Microkernel?

2006-05-16T16:35:00.000-07:00

Keith Adams has posted an entry about his thoughts on microkernels. I completely agree with his argument. He makes a curious point though at the end of the article:

Xen has been carefully, and somewhat silently transmogrifying itself into a microkernel.

What I find curious about this statement is that I think the inverse is actually true. Historically, starting in the 2.0.x series, Xen owes quite a bit of it's design to the Nemesis exokernel. A lot of the things that seem out of place in a hypervisor (event channels and usage of the nomenclature of domains) are actually vestiges of Nemesis.

Xen is not becoming a microkernel, rather it was an exokernel and is becoming more monolithic. This leads to a more interesting topic of debate (that I think Keith is partially alluding to). Do the same arguments about about microkernel vs. monolithic kernel apply to hypervisors?

I've always thought an interesting patch to Linux would be one that allowed the kernel to exist in 64mb. However, even this may be unnecessary. As VT/SVM hardware improves over time, and the cost of a switching address spaces during an exit decreases (thanks to tagged TLBs), it really stops mattering.

It's an interesting thing to think about...

Some thoughts on the Xen Virtual Framebuffer

2006-05-15T18:51:00.000-07:00

I have about a day of free time before my family comes in for graduation. I've figured I'd use some of this to do a little work on the Xen framebuffer. Unfortunately, it's very low on my priority list so I've not gotten very much time at all to work on it.

The one required task I have to complete is to bring the current code up to date and to make it use XenStore. That's pretty straight forward.

Another thing I'd like to do is finally address some of the nagging performance problems I've seen. The first fix is to implement a shadow framebuffer to reduce the size of the update regions. Instead of doing a full shadow (like I did in QEmu), I'm thinking of instead just allocating a page when a portion of the framebuffer is write faulted. I'll copy the original contents of the page to the allocated page and then when it comes time to invalidate, I'll have something kosher to compare to.

I also want to implement the dirty bitmap. I like that approach much better because it lets dom0 decide how to break the region into rectangles.

Analysis of RealVNC Vulnerability

2006-05-13T08:25:00.000-07:00

As many of you may have saw, Slashdot ran a story about a flaw found in RealVNC 4.1.1 that allowed an attacker to gain access to a password protected VNC server without any credentials.

By the time I saw this article, the site was already Slashdotted so I wasn't able to get a packet trace. The disclosure of the flaw was a bit troubling as there was no explanation of how it worked. Moreover, it sounded like it was stumbled upon by accident. My first thought was buffer overflow or integer overflow but it takes a very carefully crafted packet to exploit such flaws. Plus, VNC doesn't really have many variable length strings and certainly doesn't have any in the right parts of the protocol to explain this vulnerability.

Then it dawned on me while on campus the other day. Starting in version 3.7 of the RFB protocol, the security type negotiation was changed. It used to be that the server just sent back whatever security type it wanted to use. Common security types are rfbAuthNone (no password) and rfbAuthVnc (triple-des challenge/response). The new security type negotiation allowed the server to send back a list of security types letting the client choose which one it wanted to use.

Obviously, this makes supporting new security types more backwards compatible. A really silly mistake to make in the server would be to not check to make sure that the security type the client chooses is actually part of the original list the server supports. I looked at the RealVNC source code and sure enough, they weren't checking. What happened here is that the person who found this hole must have had a bug in his client where it sent the wrong security type (one that wasn't in the list).

This brings up a pretty interesting question. Why wasn't there a test case for this? This seems like a rather obvious thing to check for. Fortunately, this is really limited to RealVNC 4.1.x as all other VNC codebases branched off before the rewrite for 4.1.x and this problem was introduced in that rewrite. If you're using RealVNC, and you do not want password-less access to your machine, you should visit RealVNC and pick up the new version that fixes this problem.

QEMU 0.8.1 is out

2006-05-03T19:13:00.000-07:00

Fabrice posted the following today:

QEMU version 0.8.1 is available at http://bellard.org/qemu/download.html.

version 0.8.1:

  - USB tablet support (Brad Campbell, Anthony Liguori)
  - win32 host serial support (Kazu)
  - PC speaker support (Joachim Henke)
  - IDE LBA48 support (Jens Axboe)
  - SSE3 support
  - Solaris port (Ben Taylor)
  - Preliminary SH4 target (Samuel Tardieu)
  - VNC server (Anthony Liguori)
  - slirp fixes (Ed Swierk et al.)
  - USB fixes
  - ARM Versatile Platform Baseboard emulation (Paul Brook)

VNC support in QEMU CVS

2006-04-30T18:29:00.000-07:00

Fabrice just committed it this afternoon. I broke the Win32 build (whoops) but Fabrice quickly fixed it.

VNC in QEmu

2006-04-23T21:58:00.000-07:00

As I mentioned in a previous post, I have been developing a VNC patch for QEmu. QEmu is a really great platform to explore different ideas as it's entirely in userspace and therefore is somewhat sane to debug. The two ideas I've been examining are both accelerations for VNC and so far have shown promising results.

The first is what I'll refer to as subtiling. A fundamental VNC optimization is to only transmit the portions of the screen that change to the client. The easiest way to do that is to keep track of which portions of the screen change and then transmit those portions during your normal update event. In a virtualized environment, it's a bit harder to keep track of which regions change because it's just memory being written to. This means you usually end up with oversized dirty regions. This tends to result in 1-2 orders of magnitude more data being marked dirty than there really is. To eliminate this, it becomes necessary to maintain a copy of the client framebuffer (that is, the framebuffer that you think the client has). When it comes time to transmit an update, instead of blindly transmitting dirty regions, I've implemented a "subtiling" algorithm which compares against our copy of the client framebuffer to determine which regions have really changed.

The second optimization I've implemented is video-to-video blit acceleration. The basic idea is that most modern VGA hardware (and more important, the Cirrus card that QEmu emulates) implements 2d acceleration routines. In the case of the Cirrus card, this is a fast copy between video memory (optionally using one of a number of ROP operators). With the right filtering and a little magic, we can convert this video-to-video copy into a VNC CopyRect operation. I suspected that this would be a useful optimization but I had no idea how much of a difference it would make. I was surprised to learn just how many places a modern operating system makes use of accelerated copying. When you scroll a web page in internet explorer, Windows uses 2d acceleration. The result is an extremely usable interface that is awfully pleasant to use even over the network. In fact, even with a poor encoding, the interface is at least as responsive on my local LAN as a real VNC server.

Tonight I got a chance to talk to Fabrice about this patch and he seemed to be happy with it. In fact, it looks like I've got a shot of getting it in for the next release. The code is quite ugly right now but hopefully I'll have something that's much cleaner by the end of the week. Then I can start looking at reusing it for Xen so I can finally get rid of the libvncserver dependency for the VFB.

Xen, VMware, and upstream merge

2006-04-22T21:22:00.000-07:00

There's been a whole lot of articles about Xen, VMware, and the on going upstream merge effort. I just want to point out a couple things. The first is that a number of the articles are a bit misleading/speculative. The second is that there is much more common ground than the articles seem to suggest.

There are only a very small number of areas where VMI and Xen differ. Most of these differences are superficial. One thing to keep in mind is that neither set of patches is complete at all. Neither support SMP (fully), writable page tables, or management mode stuff. Both Zach and Chris tried to make this very clear in the initial posts.

There's a lot of work to do and a lot of hard problems to solve. In any case, even though neither set of patches is really complete, they are still way to big. Far too big to go into Linux all at once. Upstream merge is going to require a large number of small changes to incrementally add support for virtualization into Linux. That's what's going to be interesting over the next year. Once we figure out how to break things up, we'll start doing it.

The whole VMI vs. Xen thing is overrated. This is really just about what's the best interface for Linux. The actual underlying hypervisor doesn't really matter.

Ambiguoity in RFB (VNC) specification

2006-04-22T13:07:00.000-07:00

In the RFB protocol specification (the protocol used by VNC), the best non-compressed protocol is by far Hextile. Hextile tile's the screen and provides a mechanism to cheaply enough solid tiles, or tiles that contain subrects. One of the optimizations is that each tile can have a background and foreground color and if they aren't specified, they are implicitly assumed to be the previous tiles color.

There is a special tile mode called SubrectsColoured (or SubrectsColored for us Yankees). In this mode, each sub rectangle has its own color specified before the subrect coordinates. What the specification doesn't make clear, is that this color is treated to be a "foreground" color such that if the next tile relies on an implicit foreground color, the foreground color should be the very last colored subrectangle's color.

I wasted a good part of a day on this and most certainly much longer when I was previously working on a client. I'm going to try and request that the spec be clarified to make this very explicit.

3.0.2 is finally out

2006-04-13T21:25:00.000-07:00

After a number of problems (a couple of last minute regressions and an outbreak of the flu), we've finally cut a 3.0.2 release. Major features include SVM (Pacifica) support, a fresh 2.6.16 Linux guest kernel, XML-RPC support in the userspace tools, and lots of bug fixes and enhancements.

You can download 3.0.2 from bit torrent here.

Absolute Mouse Support in Xorg

2006-04-09T09:01:00.000-07:00

I've been reading bugzilla entries and Xorg code and finally have a pretty good idea of the general support for absolute USB mice. Windows apparently supports absolute mice natively (although both axis' have to be absolute or relative). I don't know how Windows interprets these coordinates (whether it converts them to relative movement or actually as absolute positions).

The Linux kernel supports absolute mice just fine. Of course, the Xorg I'm running on my system doesn't. Fortunately, there's a bugzilla entry against the evdev driver and initial support for absolute mouse events is present in Xorg CVS. To me, this means that it is a viable solution to this problem. It's at least as good of a solution as writing a custom Xorg input driver for Xen. It has the additional benefit of not requiring me to maintain a new X driver and I like that very much :-)

USB HID--The perfect match for virtualization?

2006-04-08T20:49:00.000-07:00

An interesting discussion has emerged on qemu-devel about a topic dear to my heart. Anyone who has done anything with input device emulation knows that mice are a big pain to virtualize as PS/2 only supports relative input coordinates. This requires the all-to-familar "input grabbing" mode that things like QEmu and VMware both require since the emulator has no idea where the mouse actually is within the guest (and therefore doesn't know when the mouse "leaves" the window).

The most accepted solution to this is to paravirtualize the input driver. This is the approach VMware takes and is the approach I'm currently taking in Xen. Another approach that I've explored is emulating a drawing tablet. These devices use absolute coordinates (as you want exactly what you draw to appear on the screen). I even implemented a proof-of-concept Wacom emulator for QEmu. Alas, these devices tend to be serial-based so you get no automatic probing on guest install. This means users have to manually configure the devices which is a no-go for a large set of our target user-base.

Someone on qemu-devel pointed out that the USB HID specification allows devices to be either relative or absolute. A compliant HID driver would therefore Just Work. Best of all, USB tends to be probed automatically so it satisfies that requirement. This got me thinking about how useful HID could be in general. Besides keyboards and mice, the HID also specifies things like USB speakers. This may be a practical way of having cross-platform plug-n-play sound. Plus, with things like USB over IP, there's an awful lot of potential for remoting these things.

News on the 3.0.2 release

2006-03-26T14:52:00.000-08:00

From Ian Pratt as seen on xen-devel:

"We were just going to sweep -unstable into -testing and call a 3.0.2 release, but it was felt that since there had been some significant changes in the previous few days letting it sit in unstable for a few days wouldn't be a bad thing."

Just in case you were wondering...

Xen 3.0.2 will be going out soon

2006-03-21T17:44:00.000-08:00

I dropped my latest (and hopefully final) version of the XML-RPC enablement for Xend this afternoon. The XML-RPC support is one of the last features to be merged before the 3.0.2 release so I expect we'll see 3.0.2 go out sometime this week.

I'm quite happy that we're actually keeping to our 4-8 week release cycle. Unfortunately, we're probably going to slip a tiny regression into the 3.0.2 release. Last week, I noticed that our block-attach code is failing. Apparently, we had some problems in our test suite that were masking the fact that it has been failing for quite some time now. block-attach isn't an extremely important feature so I don't think it's something that will hold up 3.0.2 but it would be nice to fix before releasing.

I'm pretty sure it's hotplug related. Our use of hotplug for device bring up definitely has been a pain for us. Hopefully Rusty's new drivers won't require these sort of things and we'll hopefully gain some robustness.

Xgl is now my default desktop

2006-03-04T08:51:00.000-08:00

I finally got around to figuring out why emacs wouldn't start. Now it will. The only thing left that would make things just perfect is if I could figure out how to have compiz start automatically.

Getting NetworkManager to work under Dapper

2006-03-03T23:30:00.000-08:00

Just FYI, if you have a network interface configured in /etc/network/interfaces then the new NetworkManager won't touch it.

I've been racking my brain for weeks over this one (well not really).

Some times you just have to derive it yourself

2006-03-03T10:13:00.000-08:00

I've got my shared memory QEMU interface working really well. So well, in fact, that I'm looking for the next hard thing to do. One of the neatest features of my old QEMU GTK gui was that it supported software scaling using GDK pixbuf.

GDK pixbuf is kind of annoying though because it's a client side image. The scaling code in GDK is very pixbuf specific too. Also, I never was able to figure out how to do partial updates of a scaled image without getting artifacts.

Last night I started reading about bilinear scaling. The first thing I learned is that is not really scaling, but filtering, but not really filtering, actually interpolation. Last night I finally came across a site that gave a good explaination of it in terms of graphics but lacked the math. The only sites that I found that had the math provided it in either a heavily solved form (solving for different things that I was looking for mind you) or just made no sense at all.

This morning, I decided to see about deriving this myself so that I'd actually understand it. The idea is pretty simple, given a point of unknown color that lies in the middle of four points of known color, calculate relative distance from each known point to the unknown point. Then use the distances (with the total distance of all points normalized to 1) as a factor for each known color and simply sum the known colors to obtain the unknown colors.

This may sound obtuse but it's really intuitive when you see it in action. It's just saying that the closer an unknown point is to a known point, the more that known point's color should contribute to the unknown point's color.

Know that I understand the algorithm and have an implementation, I've thought of many interesting optimizations. For instance, the amount that a known point contributes to an unknown point is orthagonal to the actual color of the known points. This means it can be computed once. This reduces the actual operations per update to 4 multiples and 5 adds per pixel. This seems like the sort of thing that I should be doing with MMX so I'll be looking into that.

The other part of this algorithm that requires computation is determining which known points surround an unknown point. The cool thing here is that this not only independent of the color of the pixels but computing the X coordinates of the known points given an unknown point's X coordinate is independent of the Y coordinates of the unknown point and vice versa. This means that it can be precompute once for a single row and then once for a single column! That requires very little memory (which is good because I'm slightly concerned that caching the color factors will take up an enormous amount of memory).

Fun stuff!

Host filesystem for Qemu

2006-02-24T23:12:00.000-08:00

I hacked together a simple host filesystem for Qemu this afternoon using fuse. This now becomes yet another project on my plate that needs cleanup and working through the submission process.

I use Qemu a lot and have some fancy scripts to automatically build disk images based on directories so I can easily test Xen and such. Of course, it's a real pain for things like Xend testing b/c I cannot just simply make a quick source code change and test again.

Even this it's a very primative hostfs (read only, single user), it's actually more than enough for what I need. I have to get my new webserver in line so I can make this stuff visible.

Touchscreen Emulation

2006-02-10T20:19:00.000-08:00

Today I implemented emulation for the Wacom Touchpad. The idea behind implementing this was that many remote display protocols wish to deal with the mouse in absolute coordinates whereas PS/2 mice report in relative coordinates. Stephen Tweedie had the brillant idea at the recent XenSummit of just emulating a Touchscreen instead of a PS/2 mouse which solves this problem since Touchpads use absolute coordinates.

After milling around, I finally got around to it today. All in all, it took a few hours going from zero knowledge to a functional emulated Wacom device. Having an X driver really helped. I talked a bit to some folks in IRC about how to expose it in QEmu so now I just have to throw a patch together.

Without the source for X, this would have easily taken me a couple weeks to do...

More framebuffer goodness

2006-02-08T22:24:00.000-08:00

Well, for starters, I've reach 100% pass rate with xm-test for the Xend XML/RPC code so I can take a break from that for a little bit while Xen settles down after the huge HVM merge. This leaves some time to work on the framebuffer again.

I talked to Ian for some time today about the framebuffer and in what form it should be merge. Ian has a very valid concern that we don't introduce things that are immediately deprecated (seeing something like the Cirrus FB emulated by the device model as a prime example of this). I think we were able to reach an agreement that as long as the framebuffer can provide good VNC performance (this means ARGB cursor, copy operations, etc.) it's good for quite a while (5 year time frame).

It was quite productive although a bit draining.

Can't get enough of previrtualization

2006-01-27T18:20:00.000-08:00

I spend a lot of my time pursuing ideas that I find interesting but do not necessarily pan out into anything. Over the past year, I've spent a lot of time looking into previrtualization. Previrtualization is an automated paravirtualization technique that uses the assembler to automatically convert sensitive instructions into more friendly versions.

The afterburning macros are pretty cool and I've explored a number of enhancements to that technique. However, it's fundamentally limited by the need to use a patched assembler. I'm currently exporing a simplistic post-processor approach that uses some interesting gas magic (without requiring a new assembler). We'll see how it works.

Wrapping up the XenSummit

2006-01-27T18:15:00.000-08:00

I still haven't blogged about the Summit. I really intend to. Perhaps I'll begin tomorrow.

Simon Crosby is a dink.

2006-01-26T17:56:00.000-08:00

He makes Xen look bad by spewing XenSource marketing crap all over the place.

Sigh.

FWIW, Keith, I think, is assuming that we switch the full page table out to provide isolated userspace/kernelspace memory environments. We don't do this. We simply bring in a PGD for kernelspace on switch. This doesn't invalidate the whole TLB so it really isn't that bad. The problem is that the kernel has to run in ring 3 to ensure that it cannot get at the hypervisor's memory because segmentation was removed from x86-64.

Update: I should point out that Simon is a nice guy overall. I'm somewhat bothered because I'm afraid that Xen does not get the respect it ought to get because there is so much FUD surrounding it in the press. Everytime someone claims that Xen is the only robust, enterprise-ready, virtualization solution we lose a little more credibility. The truth of the matter is that Xen is a young Open Source project with a lot of promise. It's not nearly as robust as other more mature projects (like the Linux kernel) and it's certainly not as robust as enterprise hypervisors like IBM's PHYP. That's not to say that it won't be in time but I think we would do better as a project if everyone involved in the project was a bit more forthright.

Almost there

2006-01-15T11:01:00.000-08:00

I'm very close to having a patch ready for xen-devel with the VFB. I just have to integrate the vncfb into the build tree and do some general cleanup.

Whew.

Major VFB refinements

2006-01-12T21:47:00.000-08:00

I've switched the XenVFB over to write-fault updating (thanks to Gerd Hoffman). Write-faulting is a method to provide partial framebuffer updates by removing write permission on the framebuffer. When a write-fault occurs, the page is brought in and a timer is activated (if not already active). When the timer goes off, all of the faulted in pages are collected up and write-protected again and an update event is triggered for the rectangle that contains all of these pages.

It's a pretty clever technique that Gerd uses for the UML framebuffer. Its going to require a bit of tweaking to get right for VNC though.

In Joisey

2005-12-23T06:42:00.000-08:00

I'm currently in Jersey and have found some time to do a little hacking. The current distraction is libxend. There are a few reasons I'm spending time on it. The first is that it's a lot of easy work which makes it ideal for a few minutes here and there of hacking. Second is that I need it for a project I'm working on at IBM. A big part of this is too is that I really want to build my own tool chain again--not a public one like vm-tools but just something that I can use. I really dislike the xm interface. I don't want to use configuration files or anything of that junk. I particularly despise the xm networking mechanism too.

A Good VM Widget

2005-12-18T10:52:00.000-08:00

There's four display modes that I care about for running virtual machines. These names are made up and probably in use by noone but me. The first mode of interest is serial mode. In this case, you assume the guest thinks you have a vt100 terminal connect to the serial port and tell the guest to display the console to that. When dealing with serial mode, you need something capable of rendering a vt100 terminal (which is no small task!).

The second, closely related, mode is CGA mode. In CGA mode, the guest thinks it has a VGA screen attached to it but since it's just displaying text, it uses the old CGA compatibility mode supported by VGA. In this mode, instead of rendering pixels to the screen, the screen is divided into characters with a few simply attributes (and a foreground/background color). There are some aspects of CGA mode that make it far superior to serial mode. Namely, the experience identical to being in front of a physical computer (scroll back works as does VT switching). A down side is that it is identical to being in front of an actually computer (so scroll back is really short and goes away on VT switching). Only some simply drawing primatives are needed to implement CGA mode (rendering a font with certain colors).

The third mode is local VGA mode. In local VGA mode, you're rendering the VGA display (in whatever depth it's setup in) to the local display. In this mode, you can do some interesting pass through of 2d or 3d drawing primatives.

The final mode is remote display mode. In remote display mode, you're on a completely different machine than the VM and the display has to be sent across the network via something like VNC.

Clearly, these four modes present interesting challenges and *no* virtualization solution implements them all well. What would be really cool to see in the future is a single GTK widget that was capable of handling any of these modes. This would help make it possible to build a really nice GUI that Just Worked either as a local or remote management interface.

A neat VFB feature

2005-12-16T22:55:00.000-08:00

A patch was recently posted on qemu-devel that allows for the VGA device (when in CGA mode) to be rendered directly by ncurses. It's a pretty neat effect and given a more complete implementation would be really cool.

In fact, I think that long term, for Xen, what I would like is to support this by default and only use the virtual framebuffer if it's enabled by the guest.

Userspace VFB Updates

2005-12-16T11:02:00.000-08:00

Gerd Knorr pointed me to a VFB driver he recently wrote for UML. The similiarities are pretty interesting and a good sign that I'm going in a plausible direction. Gerd is doing one thing in his driver that peeks my interest quite a bit.

Userspace VFB writes (ala X) are challenging because userspace gets to map the framebuffer directly into its address space. Since there is not update notification mechanism, this makes it very challenging to determine when the framebuffer should be blitted on screen (and which portions should be blitted). Its terribly wasteful to blit the entire framebuffer 30 times a second and even more wasteful to try and map that to something like VNC.

I had briefly considered using demand paging to work around this problem. Linux's mmap() is lazy in that it only maps pages when they are first read/written. You could potentially trap writes by changing the mappings to be read-only or even just not actually map the pages.

Of course, you want to avoid taking a hit for every single byte read/write. I figured this was going to be challenging and put it off. Gerd, however, has found a simple and clever approach. Gerd uses a timer that fires at the FPS rate and if there was a page that has been mapped in since then, he removes the mapping and issues appropriate updates. That's it. It's quite simple and yet also quite powerful. It has all the right properties too of little-to-no performance overhead when the display is idle. I'll certainly be adopting it for the Xen VFB.

A Bit ADD

2005-12-15T21:58:00.000-08:00

I can definitely be ADD sometimes. I haven't worked on the VFB for a few days now mostly from studying. I have some free time today which resulted in me playing around with iCal. It's pretty cool stuff.

It makes me wonder why we don't have /var/spool/calendar and proccalendar...

Xen Attracts a Crowd

2005-12-05T19:13:00.000-08:00

One of the really interesting things about the Xen project is that there are a ton of big name folks involved in it in some way or another. While this is generally an awesome thing (you get a bunch of awesome information you could get nowhere else), it sometimes means simple conversations end up becoming major holy wars.

Xen Virtual Framebuffer

2005-12-05T16:41:00.000-08:00

I posted my first version of a Xen Virtual Framebuffer today. Hopefully this will lead to lots of pretty GUIs that makes Xen as accessible to the average user as a product like VMware. It's amazing how much people love VMware. This weekend someone who I had never met before spent 10 minutes telling me how his company couldn't exist without it.

I think a large part of that is ease of use. He really didn't care if it was slow, or expensive, or whatever. It Just Works. It's a bit early for New Years resolutions, but focusing on ease-of-use is going to be mine.

If you're interested in the VFB stuff, check out http://wiki.xensource.com/xenwiki/VirtualFramebuffer.

Virtual Framebuffers

2005-12-01T20:31:00.000-08:00

One really neat thing that we do at IBM is have weekly architecture calls often with guest speakers. This combines the LTC and Research together and is often a place where really cool things are discussed.

Jimi was able to arrange for Ricardo Baratto from the NCL at Columbia to give a presentation on THINC this week. I wasn't able to attend due to class but after reading a couple of the papers on it, I grew a new interest in a different approach to para-virtual framebuffers.

Apparently, the framebuffer interface in Linux is butt-simple and I was able to hack up a pretty quick vmalloc() based framebuffer. I'm currently plumbing it up within Xen and will hopefully have a demo soon.

PyGTK VncView Wrapper

2005-11-23T23:10:00.000-08:00

It was amazingly easy once I fonud a developersWorks article on it. In a 34 line python app, I have a fully functioning VNC viewer that supports tabbing :-) Swank!

SetName VNC Encoding

2005-11-23T22:17:00.000-08:00

I decided to hold off on the zlib reset VNC encoding extension. I'm contemplating whether recompressing is an option (I want to avoid protocol extensions at all cost).

However, one extension that I know is necessary is and not terribly contraversarily is updating the name of the desktop. This extension is very simple, a new pseudo-encoding that is sent as a U32 that is the size of the new name, followed by the name.

That's it.

Extending the RFB Protocol

2005-11-23T17:33:00.000-08:00

I just realized that I needed to add a new extension to the RFB protocol to support my proxy. The problem is that each client maintains one or more zlib streams that are used to decrypt the data. These streams must be maintained used to process the data in order.

Proxying VNC sessions then can cause stream corruption because there's no way to signal the client to clear out the stream. How do we fix this?

Introduce a new encoding that the server issues to signal a switch to a new set of streams.

My old university happens to be in the most dangerous city in the US

2005-11-21T20:11:00.000-08:00

According to Reuters, Camden, NJ, the home of the Rutgers campus I attended before transfering to UT, is the most dangerous city in the country for the second year in a row. The muder rate was 10 times the national average and the robbery rate was 7 times the national average.

And people asked why I transferred...

And from nowhere, SMP support in QEMU

2005-11-21T19:57:00.000-08:00

Being on qemu-devel is like having a monthly Christmas present. Out of nowhere, Fabrice just checks in amazing things. A few weeks ago it was USB, and this evening, it was SMP support.

It's currently just guest SMP (the virtual CPUs are all multiplexed in the same process) but I wouldn't be surprised to see host SMP appear in the near future.

I wonder if SMP Xen will work under QEMU.

A really nasty bug

2005-11-21T15:53:00.000-08:00

The Tight VNC Encoding has a number of different filters to improve data compression quality. One of those filters is "palette" which converts the pixel data into an indexed 8-bit representation. Before sending the data, the size of the palette (minus 1) is sent as a single byte and then the palette is sent followed by the filtered pixel data.

I had code that looked something like this:


uint8_t palette_size = ptr[0] + 1;
uint8_t *palette = ptr + 1;
uint8_t *pixel_data = ptr + palette_size * bpp + 1;

decode_length(pixel_data);

Strangely, when attempting to decode the length, I was getting an impossibly large number which resulted in garbage being passed to zlib resulting in a decompression error. So where's the error? It's subtle. If palette_size == 256, or ptr[0] == 255, then palette_size silently overflows to 0 resulting in pixel_data actually being the palette. Curiously, I used a int for palette_size in the protocol parser so I wasn't having a problem there. Doh!

VNC client library needs some work

2005-11-21T10:55:00.000-08:00

The last real feature I needed for my vncviewer widget was Tight Encoding. As I mentioned in an earlier post, I had worked out a protocol parser so today I decided to sit down and write a decoder.

I ran into pixel format hell though as Tight uses packed RGB pixels as a special case when bits_per_pixel == 4. ZRLE uses it too but it's considerably easier to deal with in that case. This has led me to believe that it's time to factor out the decoding routines into a completely different library. Instead of providing drawing primatives, I think I really have to just give it a buffer like with QEmu.

I'm done for the night

2005-11-20T20:25:00.000-08:00

My wrists are giving out.

I just can't help myself

2005-11-20T16:51:00.000-08:00

I've been using gnome-blog-poster for my posts so far and while it's mostly nice, it uses the old XML-RPC interface for blogger which doesn't support titles. So, I've written my own that uses the new Atom based API.

Tight encoding and a new approach to proxying

2005-11-20T11:57:00.000-08:00

I finished implementing support for Tight encoding in my VNC protocol library today. This is exciting because Tight is clearly the best encoding scheme that's supported by the largest number of servers out there.

I also had an interesting thought about how this should all work within Xen. I was envisioning a single daemon that was told to open a PTY and listen on two sockets (one for the incoming connections and one for the reverse connection).

I'm now thinking that a much more generic proxy is a better answer. The idea would be to have a VNC proxy that can be given an arbitrary number of VNC connections to multiplex (that are normal or reverse). A config file of some sort could define the key bindings used for switching. The key bindings would obviously have to be directional (alt-f7 to go from session 1->2 and ctrl-alt-f1 to go from session 2->1).

I would then just implement a VNC terminal program and run that as a normal VNC server.

An interesting idea

2005-11-19T18:25:00.000-08:00

I just had a rather interesting idea. One downside of using Xvnc for a paravirtual framebuffer is that it requires a TCP connection to dom0. This is problematic for a few reasons. The first is that it assumes networking is up which is generally a big fat assumption. Imagine the use-case of a distro installation.

The second problem is that Xen supports network interface crediting which is going to be an important feature for a lot of users. A paravirtual framebuffer is going to use a lot of bandwidth so this is not practical.

While thinking about how much work it would be to port Xvnc to user a shared memory ring queue, I had a revelation. Why not create a daemon that listens on a socket in domain-U and then transfers all the data on that socket via a ring queue to then expose it (as another socket) in domain-0. You now have a dedicated shared memory transport that doesn't require networking. It should be relatively high performance (certainly better than just using a vif).

Much more elegant than something like dedicating a vif.

VNC Everywhere

2005-11-19T16:33:00.000-08:00

I've been working with VNC a lot lately. You may or may not know that I rewrote QEMU's GUI to be a native GTK application (more info here). After seeing how easy this was and how much I enjoyed it, I decided I really wanted the same thing for Xen.

This means that the virtual framebuffer problem has to be solved. It's a big one that noone's wanted to touch so far and that we get an awful lot of criticism for.

A few of the Cambridge guys want to do the obvious. Implement a paravirtual framebuffer driver in Linux. They even have some smart ideas about strifing it across pages to identify update regions. I don't like this approach though because it's going to have a noticable performance impact (event if noone's connected, we still have to render the console text to the buffer) and it's a lot of work.

I have a different approach. The idea is to leverage Xvnc (as many do right now with Xen). In domain-0, we would have a fake VNC server that connects a domain-U's emergency console and renders the console to VNC. This is essentially a VNCTerm program and not an amazingly new concept. The clever part in this story though is having this fake VNC server also play the role of a VNC proxy by setting up the domain-U's X server to actually be Xvnc configured to use reverse VNC to connect back to the fake VNC server. When X starts up in the domain, the fake VNC server would accept the reverse connection and do a virtual VT switch and begin acting just a VNC proxy.

By utilizing the DesktopSize VNC psuedo-encoding, this would give the appearance of Just Working. The fake VNC server can also trap ctrl-alt-fN key events to switch back to the emergency console.

If the distro's are willing to play ball here, this can be easily extended for graphical installers too since SUSE has had support for installations over VNC for quite some time and RedHat would simply have to launch Xvnc to run Anaconda in.

I'm somewhere between the proof-of-concept and alpha stage right now. I've written my own VNC protocol parsing library (that's a whole other post), a GTK VNC viewing widget, a VNCTerm server, and a VNC proxy (using the protocol library). Everything needs a lot of cleanup and some bug fixes. The performance is already very good so I'm quite happy about that. With any luck, Xen 3.1 will be extremely easy to use.

A little introspective

2005-11-19T15:43:00.000-08:00

I've maintained many blogs in my time but they all have been self-hosted and using software that I wrote. This meant blogging involved quite a bit of work on my part.

I've been wanting to start blogging about Xen development for some time now but have not wanted to invest the time in blogging again so I thought I'd give using blogger a go.

We'll see how I like it. For now, please use http://blog.codemonkey.ws to access this blog and you'll always go to the right place.

first post

2005-11-19T15:38:00.000-08:00

first post!