I ran across an interesting overview paper, Attacks on Virtual Machine Emulators by Peter Ferrie, Senior Principal Researcher, Symantec Advanced Threat Research.

Abstract – As virtual machine emulators have become commonplace in the analysis of malicious code, malicious code has started to fight back. This paper will explain known attacks against the most widely used virtual machine emulators (VMware and VirtualPC). It will also demonstrate newly discovered attacks on other virtual machine emulators (Bochs, Hydra, QEMU, and Xen), and describe how to defend against them.

A lot of the paper covers detection which I would say is different from an attack.

An interesting thing discussed is a way to use the CPUID instruction in combination with examining pages in the TLB to detect the presence of VMMs (cf. this previous entry here).

There is also a description of an authentication method that Parallels employs, a session key placed into the general registers by the guest (it also discusses a way of crashing Parallels on demand).

Slides and the paper can be downloaded from the author’s homepage.