Migrate KVM virtual machines from CentOS 6 to Fedora 18 without the luxury of shared storage

I’ve converted one of my KVM hypervisors from CentOS 6 to Fedora 18 and now comes the task of migrating my virtual machines off of my single remaining CentOS 6 hypervisor. This is definitely on a budget, so there’s no shared storage to make this process easier.

Here’s how I did it:

Migrate the logical volume
My first VM to migrate is my Fedora development VM where I build and test new packages. I have a 10G logical volume on the old node:

[root@helium ~]# lvs /dev/mapper/vg_helium-fedora--dev 
  LV         VG        Attr     LSize  Pool Origin Data%  Move Log Copy%  Convert
  fedora-dev vg_helium -wi-a--- 10.00g

I made a 10G logical volume on the new hypervisor:

[root@hydrogen ~]# lvcreate -n fedora-dev -L10G vg_hydrogen
  Logical volume "fedora-dev" created

After getting ssh keys set up between both hypervisors and installing pv (to track progress), I started the storage migration over ssh:

dd if=/dev/mapper/vg_helium-fedora--dev | pv | ssh hydrogen dd of=/dev/mapper/vg_hydrogen-fedora--dev

Luckily it was only a 10GB logical volume so it transferred over in a few minutes.

Dump and adjust the source VM’s XML
On the source server, I dumped the VM configuration to an XML file and copied it to the new host:

virsh dumpxml fedora-dev > fedora-dev.xml
scp fedora-dev.xml hydrogen:

Before importing the XML file on the new host, there are some adjustments that need to be made. First off was an adjustment of the storage volume since the new host had the same logical volume name but a different volume group (the source line):

<disk type='block' device='disk'>
  <driver name='qemu' type='raw' cache='none' io='native'></driver>
  <source dev='/dev/vg_hydrogen/fedora-dev'/>
  <target dev='vda' bus='virtio'></target>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'></address>

Also, there’s a mismatch with the machine type (not architecture) between CentOS 6 and Fedora 18. I dumped the XML from a VM running on the Fedora 18 hypervisor and compared the machine type to my old CentOS VM’s XML (the XML from the CentOS VM is on top):

-    <type arch='x86_64' machine='rhel6.3.0'>hvm</type>
+    <type arch='x86_64' machine='pc-1.2'>hvm</type>

I replaced rhel6.3.0 with pc-1.2. If you forget this step, your VM won’t start. You’ll get some errors about a mismatched machine type before the VM boots.

There’s one last fix: the path to the qemu-kvm emulator:

-    <emulator>/usr/libexec/qemu-kvm</emulator>
+    <emulator>/usr/bin/qemu-kvm</emulator>

Replace /usr/libexec/qemu-kvm with /usr/bin/qemu-kvm and save your XML file.

Import the VM configuration and launch the VM
Importing the VM on the Fedora 18 hypervisor was easy:

virsh define fedora-dev.xml

That causes the configuration to load into libvirt and it should appear in virt-manager or virsh list by this point. If not, double check your previous steps and look for error messages in your logs. That doesn’t actually start the virtual machine, so I started it on the command line:

virsh start fedora-dev

Within a few moments, the VM was up and responding to pings.

It’s a good idea to hop into virt-manager and verify that the VM configuration is what you expect. Some configuration options don’t line up terribly well between CentOS 6 and Fedora 18. You might need to adjust a few to match the performance you expect to see.


  1. says

    If you have

    –append=”console=tty0 console=ttyS0,115200

    in you guest’s kernel command line, you don’t even have to hop into virt-manager.

    You can do

    $ virsh start fedora-dev — console

    And you can see the boot progress on your serial console.

    If you knew all of this anyway, just disregard me :)


  2. says

    That’s very true. I usually end up in virt-manager anyways and I forget about using the serial console there on the command line.

  3. Kevin says

    I often use lzop (very fast compression) as such

    dd if=/dev/mapper/vg_helium-fedora–dev | pv | lzop | ssh hydrogen “lzop -d dd of=/dev/mapper/vg_hydrogen-fedora–dev”

    to speed up the process, especially if I know most of the content might be easily compressible or even zeroed-out.

  4. Kevin says

    I meant to add a pipe for the decompression of course:

    “[…] lzop -d | dd […]”

  5. says

    Not a good practice but i prefer to use nc when i need to move big images files over the LAN, you can even use port 22 so you won’t have to play with your filewall.

    Again, not a good practice, but faster =)

  6. says

    I wrote lvmsync (http://theshed.hezmatt.org/lvmsync/) for this exact use case. Instead of the VM having to be down for the entire period of the disk transfer, you can take an LVM snapshot then transfer the “base” image across while the VM is still running. Only then do you shutdown the VM and run lvmsync over the snapshot, and it transfers the blocks that the snapshot has recorded as having been changed.

    My personal highlight so far has been taking a 1.1TB VM and migrating it to a new machine with less than two minutes of “doesn’t respond to ping” downtime. Nagios barely even noticed!

  7. lzap says

    Heads up – ssh can be configured to compress by default in some distros. Turn it off with -O when copying. For RHEL it is turned off luckily.

Leave a Reply

Your email address will not be published. Required fields are marked *