An interesting idea
The second problem is that Xen supports network interface crediting which is going to be an important feature for a lot of users. A paravirtual framebuffer is going to use a lot of bandwidth so this is not practical.
While thinking about how much work it would be to port Xvnc to user a shared memory ring queue, I had a revelation. Why not create a daemon that listens on a socket in domain-U and then transfers all the data on that socket via a ring queue to then expose it (as another socket) in domain-0. You now have a dedicated shared memory transport that doesn't require networking. It should be relatively high performance (certainly better than just using a vif).
Much more elegant than something like dedicating a vif.