Classical Virtualisation

Pure virtualisation doesn't work on modern architectures, because processors such as the Pentium and the Itanium have instructions that modify or access system state, but do not trap. In addition, the cost of trapping on each such instruction means that the guest kernel runs very slowly.

One approach to removing the overhead is to replace sensitive instructions in the guest with sequences that emulate them either by manipulating a virtual CPU in memory, or by calling into the supervisor by some low-cost means.

However, manually finding and replacing all such sequences is tedious and error-prone, and leads to large, hard-to-maintain patches.

There are two alternatives: binary rewriting (as used in VMware) and Afterburning.

Afterburning

When building a linux Kernel, the assembler already knows how to recognise sensitive instructions in order to convert them to binary. Commmonly used toolchains (e.g., gcc) produce ascii-format assembly language as one of their stages.

If we replace the assembler with our own script, we can modify the sensitive instructions in the instruction stream before the real assembler sees it. This is the essence of afterburning. For the Itanium afterburner, MatthewChapman has written a perl script that does exactly this. It opens a pipe to the real assembler, then translates each assembly language line using a set of regular expressions. A sample of the regular expression below shows how the sensitive instructions get replaced by macros defined by the afterburner script. 

my $start = '(?:\((p[0-9]+)\)[[:space:]]*)?\b';
my $end = '\b';
my $space = '[[:space:]]+';
my $equals = '[[:space:]]*=[[:space:]]*';
my $comma = '[[:space:]]*,[[:space:]]*';
my $reg = '((?:r|in|out|loc)\d+|sp|gp)';
my $index = "[[:space:]]*\[[[:space:]]*${reg}[[:space:]]*\]";
# cover
s/${start}cover${end}/emul_cover pr=$1/g;

# rfi
s/${start}rfi${end}/emul_rfi pr=$1/g;

# fc
s/${start}fc${space}${reg}${end}/emul_fc pr=$1,src=$2/g;

# thash/ttag/tpa
s/${start}(thash|ttag|tpa)${space}${reg}${equals}${reg}${end}/emul_$2 pr=$1,dst=$3,src=$4/g;
...

The emul_xxx macros are included in an assembly source file that is included in every assembly file by the afterburning script. The resulting assembly code that is finally handed to the real assembler contains sensitive instructions substituted with calls to the macros. These macros are responsible for ensuring that the supervisor state is correctly updated to reflect the corresponding change in the guest kernel.

Pre-virtualisation

(also known as Pre-virtualization) (We do not currently do this for Itanium)

One of the problems with Pre-virtualisation or binary rewriting on Itanium is that, as it has a register-based RISC-like instruction-set architecture, it is impossible to do anything unless you can use a register. In addition, the bundling requirements mean that code expansion can be a problem, as stops and bundles can move into the wrong places.

One way to ensure that there is enough room is to insert extra NOP instructions after each sensitive instruction, but leave the original instruction in place. Also, add a table of addresses for each sensitive instruction. Then the hypervisor at load time can rewrite the instruction stream.

Projects

Afterburning technique is used by the following projects:

This page will deal mostly with the Afterburner implemented for Xen on Itaniums.

Xen-IA64

The Xen-IA64 afterburner is being updated to run with latest Xen and Linux kernels. Below are instructions to build xen and linux for afterburning. The result does not work (yet). Watch this space!!!

  1. Check out xen revision 17132. 
     $ hg clone -r17132 http://xenbits.xensource.com/ext/xen-ia64-unstable.hg ./xen17132

  2. Download the xen afterburner patch. This patch makes minor modifications to enable xen to run an afterburnt kernel. Click the following link: xenab_xenrev17132.patch. Once this patch has been applied, make sure that /usr/bin/ia64-linux-gnu-as is a link to the assembler for ia64 architecture. Create the link, if it doesn't exist. This is needed because the pseudo-assembler or the perl script used for afterburning, calls this path after making the appropriate substitutions in the assembly code handed to it by gcc.

  3. Apply patch. Inside Xen source root, run the following command. 
     $ patch -p1 < xenab_xenrev17132.patch

The above patch also contains the scripts and assembly code required for afterburning a Linux kernel. These are put into a directgory called afterbuner-xen. Make sure that afterburner-xen/as is executable:

(the afterburner-absolute-path is the absolute path to the afterburner-xen directory, created by the patch). You'll need to change the permissions on afterburner-xen/as so that it is executable. This is needed so that gcc can pipe the assembly output to the afterburning script, which replaces the sensitive instructions with virtualized instructions.

  1. The above build process will fail with the following error message. 
     make[1]: *** No rule to make target `/usr/src/shehjart/linux-2.6.24.4/afterburner-xen//burn-gate.o', needed by `arch/ia64/kernel/gate.so'.  Stop.

  2. At this point, we need to build the afterburner-xen/burn-gate.S using the same command with which make tried to build arch/ia64/kernel/gate.so. This command will be the last gcc command in make output. Since we've specified V=1 argument to make, the last gcc command will be visible on the terminal. In that command, replace all mentions of the file being built, with afterburner-xen/burn-gate.o.

  3. The Linux kernel build can then be restarted using the same build command above, once burn-gate.o has been built.

  4. Build xen. To build xen with afterburning enabled, edit the Config.mk and add the following line. 

     CONFIG_AFTERBURN := y
    There are also incompatibilities with recent GCC. I added 
      HOSTCC := gcc-4.1
  CC := gcc-4.1
    Next, build only the image for xen by: 
     $ cd xen; make

    In all future builds, simply copy the dist/install/boot/xen-3.3-unstable.gz into the boot partition and run elilo.

  5. After both xen and afterburnt linux have been built, copy the zipped images into the boot partition and update elilo as shown at PaulDavies/XenIA64Howto. Make sure that the images for patched xen and the afterburnt Linux are named differently. We dont want any working images in /boot to get clobbered by elilo.

IA64wiki: AfterBurning (last edited 2008-06-25 02:39:57 by PeterChubb)

Gelato@UNSW is sponsored by
the University of New South Wales National ICT Australia The Gelato Federation Hewlett-Packard Company Australian Research Council
Please contact us with any questions or comments.