There's four display modes that I care about for running virtual machines. These names are made up and probably in use by noone but me. The first mode of interest is serial mode. In this case, you assume the guest thinks you have a vt100 terminal connect to the serial port and tell the guest to display the console to that. When dealing with serial mode, you need something capable of rendering a vt100 terminal (which is no small task!).
The second, closely related, mode is CGA mode. In CGA mode, the guest thinks it has a VGA screen attached to it but since it's just displaying text, it uses the old CGA compatibility mode supported by VGA. In this mode, instead of rendering pixels to the screen, the screen is divided into characters with a few simply attributes (and a foreground/background color). There are some aspects of CGA mode that make it far superior to serial mode. Namely, the experience identical to being in front of a physical computer (scroll back works as does VT switching). A down side is that it is identical to being in front of an actually computer (so scroll back is really short and goes away on VT switching). Only some simply drawing primatives are needed to implement CGA mode (rendering a font with certain colors).
The third mode is local VGA mode. In local VGA mode, you're rendering the VGA display (in whatever depth it's setup in) to the local display. In this mode, you can do some interesting pass through of 2d or 3d drawing primatives.
The final mode is remote display mode. In remote display mode, you're on a completely different machine than the VM and the display has to be sent across the network via something like VNC.
Clearly, these four modes present interesting challenges and *no* virtualization solution implements them all well. What would be really cool to see in the future is a single GTK widget that was capable of handling any of these modes. This would help make it possible to build a really nice GUI that Just Worked either as a local or remote management interface.