Preparing custom images for OpenStack

This article will show you how to use libvirt to create base images that can be uploaded to OpenStack.

Why would you want to do this?

Linux distributions like Fedora and Ubuntu already ship "cloud" images and most providers also have their own custom images for you to use, but I find it much more comforting to have full control of the software that is installed and I like the ability to easily apply new security patches to base images.

I wouldn't use images to replace config management (CM) with something like Salt or Ansible but they are nice to give sane system defaults in things like grub.conf, sysctl.conf, and shipping a Chef or Salt agent so that your CM engine can communicate with your server right away.

Setting up your environment

The first thing you need to do is get a minimal install disk for the Linux distribution you want to use. I prefer using Fedora netinst disks but another popular option is Ubuntu Server.

To get the latest Fedora here, you can choose "netinst" under Direct Downloads: http://fedoraproject.org/en/get-fedora-all

To get the latest Ubuntu you can go here: http://www.ubuntu.com/download/server

Once you have acquired your distribution of choice you just need to verify that you have virt-install and virt-viewer installed:

Fedora:

yum install virt-install virt-viewer

Ubuntu:

apt-get install virtinst virt-viewer

If you prefer a graphical user interface, you may use virt-manager instead, but I try to keep everything in the CLI; that way it can be repeated easily.

Preparing your disk

Now that you have a base ISO and the tools necessary, let's get started by creating a disk to install the virtual server into. Resizing an image isn't an impossible task but it is much easier to choose a reasonably sized disk for the purpose it will be used for.

I primarily use 8 GB disks -- that way we can fit all the system components required as well as our own web applications. Any large files should be placed in a SAN or something like Dreamhost's dreamobjects.

The other big decision you must make upfront is what disk format you want to use -- the trade-off is disk space vs performance. The two primary formats are qcow2 (QEMU Copy on Write) and Raw. qcow2 is great if you have limited disk space and don't want to allocate the full 8 GB up front. Raw is preferred if you want the best performance.

If you choose qcow2, you'll also need to make sure you have qemu-img:

Fedora:

yum install qemu-img

Ubuntu:

apt-get install qemu-utils

Create a raw disk:

fallocate -l 8192M server.img

Create a qcow2 disk:

qemu-img create -f qcow2 server.qcow2 8G

Installing your distribution onto the disk

We will use the virt-install command to get the distribution installed onto the disk image.

To install Fedora on a qcow2 disk image:

virt-install --name base_server --ram 1024 --cdrom=./Fedora-20-x86_64-netinst.iso \
--disk path=./server.qcow2,format=qcow2

To install Ubuntu Server on a raw disk image:

virt-install --name base_server --ram 1024 --cdrom=./ubuntu-12.04.4-server-amd64.iso \
--disk path=./server.img,format=raw

You should follow the standard install steps that you normally would when setting up your distribution. But here are some tips for each:

Fedora:

  • Choose minimal install -- by default it selects "GNOME".

Ubuntu:

  • Be sure to select OpenSSH server -- it won't install it by default.

  • On Ubuntu 12.04, there is a bug that makes it hang after running fsck. You will need to edit grub to get it to boot, hit _e_ at the boot prompt and add "nomodeset" on the linux line. You will know that you need to do this if your boot hangs on fsck:

    fsck from util-linux 2.20.1
    /dev/mapper/ubuntu--vg-root: clean, 57106/441504 files, 286779/1764352 blocks
    /dev/sda1: clean, 230/62248 files, 39833/248832 blocks
    

Preparing image for openstack

To prepare a virtual machine for the cloud, you will need to install the cloud-init package, which allows the cloud providers to inject certain system settings when creating servers based on the image. These are things like hostname and ssh keys.

On Fedora:

yum install cloud-init

On Ubuntu:

apt-get install cloud-init

Then you need to just configure cloud-init by editing /etc/cloud/cloud.cfg and update the datasources_list section to include EC2. OpenStack uses EC2 metadata for cloud-init.

You should also verify the user setting in this same config and define the user you plan to use, it will be where the authorized_keys file is setup for when the cloud provider injects your SSH key into the server.

cloud-init will not create the user for you, it will just assign the SSH keypair and reset the password. So make sure the user defined in cloud.cfg is also created on the system.

Once you have your cloud-init settings the way you want them, just shutdown and run the virt-sysprep command.

On the guest machine:

shutdown -h now

On the host machine:

virt-sysprep -d base_server

Uploading your image to OpenStack

Using the glance API it is very straightforward to upload the image to OpenStack. Just run the following command:

glance image-create --name base_server --disk-format=qcow2 \
--container-format=bare --is-public=True --file server.qcow2 --progress

Once the image upload completes you will be able to use it immediately within nova. You can reference it by name or by the id from glance image-list.

To create your first instance from the image:

nova boot --flavor m1.tiny --image base_server --key-name devops \
--security-groups free_for_all test_server

Obviously the security groups, key name, and flavors are based on your installation of OpenStack but can all easily be queried from the nova API:

nova flavor-list
nova secgroup-list
nova keypair-list

And you are done! You'll be able to re-use your new image as a base for all new instances you spin up in openstack!