"Gentoo" DomU in XenServer

7

Overview of Xen

If you’re here, odds are you’ve heard of virtualization and, specifically, xen. Xen is an amazing piece of technology. So amazing that a company named Citrix has created a product sourced from it: XenServer.

As an additional bit of background, I love the Linux distro: Gentoo. Gentoo is distinguished because of it’s package system: portage. Wouldn’t it be nice if one could combine the amazing Xen with the amazing Gentoo? Turns out, it’s really hard.

One of the primary draws to Xen is the ability to do what’s called a migration. And to boot, the ability to do a live migration, that is, moving a virtual machine from one physical host, to another without powering the virtual machine down. This has enormous consequences for maintenance of physical hardware. Why schedule downtime when you can avoid it all together?

So Xen, the hypervisor, provides drivers to the guest instance (also known as the domU (for unprivileged domain, or domain… unprivileged)). While Xen can host just about any operating system, in order to take advantage of the special features of XenServer, such as memory usage statistics and, of course, migrations (including live migrations), you need to have a paravirtualized operating system (PV).

With a paravirtualized operating system, the hypervisor (that’s Xen remember), boots the guest operating system and installs a special device and a place to put hypervisor information as follows:

  • /proc/xen
  • /sys/hypervisor

Goal

So the goal should be clear by now: make Gentoo Linux a paravirtualized guest domU in Citrix XenServer5.5.

How to do it

Warning

Remember, it’s not easy. You should know how to handle Gentoo and XenServer.

Also: this is based on my notes of my experience. I am not responsible for any damages frustration or any other liabilities incurred by using the steps in this article. I am not responsible for how this information is used (or not used).

This “guide” is extremely raw. Only the major steps are covered and some details have been left out. This is not a step-by-step, testedly proven guide. It’s assumed you know how to edit files and navigate your way through a kernel compilation and installation, most of which you can get from the Gentoo Handbook, though.

The good news, I have created working paravirtualized instances of Gentoo. If that’s your goal, you might pick something up.

Requirements

You’ll need to have working before you start with the remainder of this guide:

  • XenServer5.5 installed on 2 physical servers
  • XenServer5.5 configured with shared storage, such as CIFS (yuck), NFS, or, even more special, iSCSI
  • XenCenter with the above XenServer’s in a server pool (Yes, sadly, this one requires a Windows operating system)

The Overview

  1. Setup your shared storage
  2. Install XenServers
  3. Install XenCenter (but you’ve done this already)
  4. Install all networking as per your requirements ;-)
  5. Get a Gentoo install CD (I guess any install CD will do)
  6. Create your VM
  7. Install Gentoo
  8. Get the Xen linux kernel
  9. Compile your kernel
  10. Install the bootloader
  11. Configure /etc/fstab
  12. Install Citrix Xen Tools
  13. Configure the VM in Xen

Setup our shared storage, XenServers and XenCenter

I’m not covering this. Citrix has plenty of documentation and, frankly, I didn’t read most of it. Just install it and read the docs when you have questions about options. This is by far, the easiest part.

Get a Gentoo install CD

Now, you don’t NEED a Gentoo-branded install CD. I used it as I followed the directions at the Gentoo site. You can find a Gentoo install CD here:

http://www.gentoo.org/main/en/where.xml

Keep your arch in mind. Mine was EM64T (aka: Intel64, AKA: amd64) 1, which means I had to download the amd64 architecture version. If you don’t pick this, you’ll have to cross compile and that’s just ugly.

So. Go ahead and download it from a mirror near you. No need to burn it to a CD/DVD. XenCenter can mount ISO’s directly via the ISO Library shared storage. If you don’t want to set that up (but WHY? It’s so useful!), then go ahead and burn your precious CD and then pop it into the server.

Provision the Virtual Machine

From within XenCenter, provision a new virtual machine from the “Other” template. Create a 10GB virtual disk. Believe me, 10GB is pretty much the minimum.

Mount the Gentoo install CD as the CD drive in the new virtual machine.

Fire up your virtual machine

Now, follow the directions to install Gentoo until you reach the Kernel part. Remember, pick the install guide for your architecture. The guides are mostly the same (with a few differences) anyway. Remember, install everything as described in the manual until you get to the “Configuring the Kernel” part.

Get the Xen kernel

Now, when I said I wanted Gentoo, I really want Gentoo’s package manager. In order to create a PV (paravirtualized, remember) kernel, you need the Xen drivers. OOOOOK. So you need to go get those. But the only way to do that is to use a Xen-enabled Kernel. Bottom line, you can’t use the gentoo-sources kernel. I tried, but I couldn’t get it to recognize the paravirtualized devices.

So, let’s download the Xen-kernel from xen.org. Download it into your chrooted Gentoo install. You can find the kernel at http://xen.org/download/ Snag the “Linux 2.6.18 with Xen 3.4.x support source tarball” or later version if available. I highly recommend wget.

cd /usr/src/
wget "http://bits.xensource.com/oss-xen/release/3.4.0/linux-2.6.18-xen-3.4.0.tar.gz"
gzip -d linux-2.6.18-xen-3.4.0.tar.gz
tar -xf linux-2.6.18-xen-3.4.0.tar
rm linux-2.6.18-xen-3.4.0.tar
ln -s linux-2.6.18-xen-3.4.0 linux
cd linux

Now it’s time to configure your kernel. I’m not sure why, but there are TONS of extraneous options. You really only need the Xen options. Just do a search for Xen and make sure they all say “y.” However, since we’re building a domU instance, you don’t need any that say: “backend.” You must configure your processor options, however. Pick your architecture type, configure your SMP options. You really don’t need the Kernel Hacks. I said no to them all. Be sure to configure your file-system drivers. If you used the Gentoo instructions (as you should have thus far), you’ll need to include ext3 support. You do not need ext2 support, but I recommend it (at least as a module) as you’ll need to edit your boot partition if anything is wrong. Once you’re done with that, go ahead and compile your kernel.

make && make modules_install

You may now resume the Gentoo install instruction until you reach the part about bootloaders.

Install the Bootloader

Now that you’re here for bootloaders, I’ll tell you right off that it’s pretty much the same as the Gentoo installation instructions. There’s a few caveats with the kernel command. Follow the instructions from the Gentoo installation, but use the following for the kernel line in /boot/grub/grub.conf (aka: menu.lst):

kernel /boot/YOURKERNELFILE root=/dev/xvda3 xencons=tty console=tty0

So that says that you installed your kernel at /boot/YOURKERNELFILE (from the Gentoo kernel installation instructions, you should have copied your compiled kernel from arch/x86_64/boot/vmlinuz to /boot/YOURKERNELFILE. Of course, you probably didn’t name it YOURKERNELFILE and named it after what it is, like: xen3.4.0-linux2.6.18.

The root= should point to the file system where your root “/” directory is stored. You have to use the device path: “/dev/xvda3” and not /dev/hda or /dev/sda (first, they won’t exist and if they do, will cause your file systems to corrupt). Consequently, “/etc” needs to be readable. Xencons and console is required to tell XenServer how to talk with the PV guest2.

Go ahead and finish the Gentoo install. Before you reboot, see the next step in this article

Configure /etc/fstab

Your disks will appear as /dev/xvdAN where A is the letter of the “disk” which is really your shared storage as provided by XenServer. N is the partition number. You should have 3 partitions if you followed the Gentoo instructions. Make sure you edit your /etc/fstab to use those devices rather than /dev/hdAN, and if you use /dev/sdAN, you’ll cause your disks to encounter errors when doing lots of writing. Believe me. You’ll notice that your disks will re-mount as read-only. If you see this, STOP using /dev/sda! /dev/hda works, but again, to fully paravirtualize, use /dev/xvda.

Go ahead and finish the Gentoo installation. Go all the way until you have to reboot. Don’t reboot yet.

Install Citrix Xen Tools

First, download the xen source from xen.org: http://bits.xensource.com/oss-xen/release/3.4.1/xen-3.4.1.tar.gz

Download that file by:

cd /usr/local/src
wget "http://bits.xensource.com/oss-xen/release/3.4.1/xen-3.4.1.tar.gz"
gzip -d xen-3.4.1.tar.gz
tar -xf xen-3.4.1.tar
rm xen-3.4.1.tar
cd xen-3.4.1
make tools

You’ll see errors about firmware. Ignore them. You only care about tools/xenstore. And by the time you see the firmware errors, it will be built (or should be at the time of this writing).

Copy the xenstore executable to /usr/sbin
Copy all the libxenstore.so.* files to /usr/lib

cd tools/xenstore
cp xenstore /usr/sbin
cp libenstore.so* /usr/lib

Now, mount the XenTools ISO provided by XenServer (it’s a “CD” provided by XenServer. Access it via the VM configuration the XenCenter console).

Before we begin, you must emerge the RPM utility. Yes, this Red Hat’s package management, but Citrix does not yet provide an ebuild. There is a Gentoo port for RPM though, so we’ll use that:

emerge rpm

Once that finishes, navigate the XenTools CD to the Linux directory. There you’ll see the utilities. Unpack the one in the Linux folder:

rpm -i --nodeps xe-guest-utilities-5.5.0-458.x86_64.rpm

—nodeps is required as we have all the dependencies, but we don’t want the RPM system to manage our software.

When the package is unpacked, you’ll see a new script in /etc/init.d/xe-linux-distribution

rc-update add xe-linux-distribution default

to run this at boot, but first, we need to fix it.

You’ll have 3 more script files in /usr/sbin:

  • xe-daemon
  • xe-linux-distribution
  • xe-update-guest-attrs

Edit xe-linux-distribution. Add the following function below the identify_lsb() function (don’t put it IN the function, put it BELOW the function).

identify_gentoo()
{
	gentoo_release="$1"
	if [ ! -e "${gentoo_release}" ] ; then
		return 1
	fi
	distro="gentoo"
	eval $(cat ${gentoo_release} | awk '{ print "release=" $5 }' )
	if [ -z "${release}" ] ; then
		return 1
	fi
	eval $(echo $release | awk -F. -- '{ print "major=" $1 ; print "minor=" $2 }' )
	if [ -z "${major}" -o -z "$minor" ] ; then
		return 1
	fi
	write_to_output "${distro}" "${major}" "${minor}" "${distro}"
}

(Editor’s note: Line 12 edited, added a space after “—”)

Next, you should add the following line

identify_gentoo "/etc/gentoo-release" && exit 0

put it just below the following line:

identify_debian /etc/debian_version && exit 0

That’s it. It should work. The /etc/gentoo-release file has the details about the Gentoo kernel you’re using. All this script does is read that file and print out the details in a format other Citrix tools use. Shutdown the Gentoo VM and move the next step in this article.

/etc/init.d/shutdown

Configure the VM

Make sure the Gentoo VM is powered down. Go to a XenServer console (accessible from XenCenter). This is different from a VM console. You access this by selecting a server and then the console tab. Any server will do. The XenVM database is shared in XenCenter (if you have shared storage that is, which you need for this article).

You’ll have to get the uuid of the Gentoo VM first. You can see a list by doing:

xe vm-list

Record the uuid for future reference. Next, you need to set the VM boot-up options. This let’s Xen boot your kernel. Exciting right!?

xe vm-param-set uuid=UUID PV-kernel=/boot/kernel PV-bootloader=pygrub PV-args=xencons=tty HVM-boot-policy=""

(Editor’s Note: uuid is not a flag, but an argument)

The above sets the kernel for the bootloader, sets the xen bootloader for the domU kernel (gentoo) and clears the HVM-boot-policy (allows for ordering the disks before boot, we don’t want this as this prevents Xen from booting your machine paravirtually).

Now, make the disk created for this VM bootable.

xe vm-disk-list vm=NAMEOFVIRTUALMACHINE | less

You’ll see a list of disks. The disk type will be a VDI. Record the uuid for future reference. Hit “q” to exit less Now, make the disk bootable:

xe vbd-param-set bootable=true uuid=UUIDOFPREVIOUSSTEP

You may now boot the machine as a PV and it should start without issue. If you need to troubleshoot, create a second VM (using the “other” template) and boot from the Gentoo CD. Detach the storage from the server you’re configuring (that doesn’t work) and attach it to the new “Rescue” VM. You can mount the drives and chroot to configure the operating system environment, if necessary. YOu can even re-build the kernel. However, you cannot yet migrate the VM, that is until you install Citrix’s tools.

You should now be able to see memory usage, network interfaces, migrate the machine, and suspend the machine, once you start it that is.

Conclusion

That’s about it. Your Gentoo domU guest should not only be restartable, but migratable! I reconstructed these instructions based on what I did to get one guest instance built. I may have missed a few things.

Disclaimer

Citrix is the Trade Mark of Citrix Systems, Inc. I am not affiliated with Citrix Systems, Inc. in any way.

Remember, I’m not responsible for your use or misuse of the information in this article.

Lack of documentation

When trying to do this myself, I was and still am stymied by the lack of documentation about paravirtualization kernels. Even Xen.org is pretty useless when it comes to documentation. They cover how to make a dom0 (which is important, I admit), but fail to cover how to construct that kernels for the ultimate goal of Xen: running virtual machines.

There is nothing from Citrix about xenbus or xenstore. This entire guide was created from trial and error. Believe me, the bootloader errors are not fun to deal.

1 Yeah, confusing. Ain’t it?

2 If you see a on Gentoo domU boot about the Kernel Freeing Unused Memory and then seemingly stalling, odds are, you forgot to tell Xen about the console. Your OS is running, you just can’t speak with it, which makes it useless.

Comments

  1. Cássio Lange said 20 days later:

    I did something similar, the biggest difference was i used the kernel distributed by citrix.
    All functions was working in gentoo domU

  2. Roberto Rosa said about 1 month later:

    Great post, dude – it was very helpful. So, I wish to share some improvements. Instead of getting the kernel from xen, I used Gentoo’s “xen-sources”, which already have all xen patches, plus some gentoo specific patches too.

    Also, on line 12 of your “identify_gentoo” function, you have to put a space between the “—” and the “’{”, leaving it as:

    eval $(echo $release | awk F. - ’{ print “major=” $1 ; print “minor=” $2 }

    And when you add the line:

    identify_gentoo “/etc/gentoo-release” && exit 0

    to the “xe-linux-distribution” script, you don’t need to put the quotes, and so the line can be:

    identify_gentoo /etc/gentoo-release && exit 0

    And finally: when you configure the VM, in Xenserver, you don’t need the parameter “PV-kernel=/boot/kernel”. Also, the “console=tty0” argument seems to be important. This way, the line could be:

    xe vm-param-set -uuid=UUID PV-bootloader=pygrub PV-args=“console=tty0 xencons=tty” HVM-boot-policy=

    Thanks for you hard research. I hope you don’t mind about the improvements… ;)

  3. Adrian Videanu said 2 months later:

    Hi, nice tutorial,
    one small correction to identify_gentoo() (i have bash here)
    function :

    identify_gentoo()
    {
    gentoo_release=“$1”
    if [ ! -e “${gentoo_release}” ] ; then
    return 1
    fi

    distro=“gentoo” eval $(cat ${gentoo_release} | awk ‘{ print “release=” $5 }’ ) if [ -z “${release}” ] ; then return 1 fi eval $(echo $release | awk -F. ‘{ print “major=” $1 ; print “minor=” $2 }’) if [ -z ${major} -o -z $minor ] ; then return 1 fi write_to_output “${distro}” “${major}” “${minor}” “${distro}”

    }

    and
    xe vm-param-set -uuid=UUID PV-kernel=/boot/kernel PV-bootloader=pygrub PV-args=xencons=tty HVM-boot-policy=""

    uithout – before uuid

    xe vm-param-set uuid=UUID PV-kernel=/boot/kernel PV-bootloader=pygrub PV-args=xencons=tty HVM-boot-policy=""

  4. Sander said 2 months later:

    It took me half a day to figure it out, but it works now. Thanks for this.

  5. Christopher Richard Wojno said 3 months later:

    Thanks for the tips guys (2 & 3). I’ve integrated some of your suggestions.

    @4: It took me a LONG time to figure this out from scratch. Again, no one has a guide for this out in the wild.

    @1: I’d like to try it with Gentoo-Sources. I was never able to get it to work, however. I was unable to get the kernel to build the vmlinuz file for amd64 (my platform). You were probably able to do it because you were not using that platform, no? I was sure XenServer only worked with Intel/AMD 64-bit platforms.

    Thanks again for your input. I’m glad people are benefiting from my suffering trials.

  6. Bruno Lustosa said 3 months later:

    Hello, line 13 of your identify_gentoo functions is lacking double quotes just before ${major}. It should be:

    if [ -z “${major}” -o -z “$minor” ] ; then

    Excelent howto, got it up and running in no time.
    In the kernel configuration, though, you don’t need to check everything related to Xen. In fact, you really don’t want Privileged Guest (dom0), and you also won’t need the backend support stuff, only frontend.

  7. Christopher Wojno said 6 months later:

    @6: You’re absolutely correct. I noticed it a few days ago when updating XenServer Tools using Citrix’s RHEL rpm file. I’ve edited the article.

    Good catch!

    For your second part, I stated exactly that in the article. It’s just one sentence, but it’s there.

(leave url/email »)

   Comment Markup Help Preview comment