Migrate KVM virtual machines from CentOS 6 to Fedora 18 without the luxury of shared storage

I’ve converted one of my KVM hypervisors from CentOS 6 to Fedora 18 and now comes the task of migrating my virtual machines off of my single remaining CentOS 6 hypervisor. This is definitely on a budget, so there’s no shared storage to make this process easier.

Here’s how I did it:

Migrate the logical volume
My first VM to migrate is my Fedora development VM where I build and test new packages. I have a 10G logical volume on the old node:

[root@helium ~]# lvs /dev/mapper/vg_helium-fedora--dev 
  LV         VG        Attr     LSize  Pool Origin Data%  Move Log Copy%  Convert
  fedora-dev vg_helium -wi-a--- 10.00g

I made a 10G logical volume on the new hypervisor:

[root@hydrogen ~]# lvcreate -n fedora-dev -L10G vg_hydrogen
  Logical volume "fedora-dev" created

After getting ssh keys set up between both hypervisors and installing pv (to track progress), I started the storage migration over ssh:

dd if=/dev/mapper/vg_helium-fedora--dev | pv | ssh hydrogen dd of=/dev/mapper/vg_hydrogen-fedora--dev

Luckily it was only a 10GB logical volume so it transferred over in a few minutes.

Dump and adjust the source VM’s XML
On the source server, I dumped the VM configuration to an XML file and copied it to the new host:

virsh dumpxml fedora-dev > fedora-dev.xml
scp fedora-dev.xml hydrogen:

Before importing the XML file on the new host, there are some adjustments that need to be made. First off was an adjustment of the storage volume since the new host had the same logical volume name but a different volume group (the source line):

<disk type='block' device='disk'>
  <driver name='qemu' type='raw' cache='none' io='native'></driver>
  <source dev='/dev/vg_hydrogen/fedora-dev'/>
  <target dev='vda' bus='virtio'></target>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'></address>
</disk>

Also, there’s a mismatch with the machine type (not architecture) between CentOS 6 and Fedora 18. I dumped the XML from a VM running on the Fedora 18 hypervisor and compared the machine type to my old CentOS VM’s XML (the XML from the CentOS VM is on top):

-    <type arch='x86_64' machine='rhel6.3.0'>hvm</type>
+    <type arch='x86_64' machine='pc-1.2'>hvm</type>

I replaced rhel6.3.0 with pc-1.2. If you forget this step, your VM won’t start. You’ll get some errors about a mismatched machine type before the VM boots.

There’s one last fix: the path to the qemu-kvm emulator:

-    <emulator>/usr/libexec/qemu-kvm</emulator>
+    <emulator>/usr/bin/qemu-kvm</emulator>

Replace /usr/libexec/qemu-kvm with /usr/bin/qemu-kvm and save your XML file.

Import the VM configuration and launch the VM
Importing the VM on the Fedora 18 hypervisor was easy:

virsh define fedora-dev.xml

That causes the configuration to load into libvirt and it should appear in virt-manager or virsh list by this point. If not, double check your previous steps and look for error messages in your logs. That doesn’t actually start the virtual machine, so I started it on the command line:

virsh start fedora-dev

Within a few moments, the VM was up and responding to pings.

It’s a good idea to hop into virt-manager and verify that the VM configuration is what you expect. Some configuration options don’t line up terribly well between CentOS 6 and Fedora 18. You might need to adjust a few to match the performance you expect to see.

Tagged , , , , , ,

Handling terminal color escape sequences in less

This post is a quick one but I wanted to share it since I taught it to someone new today. When you have bash output with colors, less doesn’t handle the color codes properly by default:

$ colordiff chunk/functions.php chunk-old/functions.php | less
ESC[0;32m22a23,27ESC[0;0m
ESC[0;34m>       * Load up our functions for grabbing content from postsESC[0;0m
ESC[0;34m>       */ESC[0;0m
ESC[0;34m>      require( get_template_directory() . '/content-grabbers.php' );ESC[0;0m
ESC[0;34m> ESC[0;0m

Toss in the -R flag and you’ll be able to see the colors properly (no colors to see here, but use your imagination):

$ colordiff chunk/functions.php chunk-old/functions.php | less -R
22a23,27
>        * Load up our functions for grabbing content from posts
>        */
>       require( get_template_directory() . '/content-grabbers.php' );
> 
>       /**

The man page for less explains the feature in greater detail:

-R or --RAW-CONTROL-CHARS
       Like -r, but only ANSI "color" escape sequences are output in "raw" form.  Unlike -r, the screen appear-
       ance is maintained correctly in most cases.  ANSI "color" escape sequences are sequences of the form:
            ESC [ ... m
       where the "..." is zero or more color specification characters For  the  purpose  of  keeping  track  of
       screen  appearance,  ANSI  color escape sequences are assumed to not move the cursor.  You can make less
       think that characters other than "m" can end ANSI color escape  sequences  by  setting  the  environment
       variable  LESSANSIENDCHARS to the list of characters which can end a color escape sequence.  And you can
       make less think that characters other than the standard ones may appear between the ESC  and  the  m  by
       setting the environment variable LESSANSIMIDCHARS to the list of characters which can appear.
Tagged , , , , , ,

Changing your ssh server’s port from the default: Is it worth it?

Changing my ssh port from the default port (22) has been one of my standard processes for quite some time when I build new servers or virtual machines. However, I see arguments crop up regularly about it (like this reddit thread or this other one).

Before I go any further, let’s settle the “security through obscurity” argument. (This could probably turn into its own post but I’ll be brief for now.) Security should always be applied in layers. This provides multiple levels of protection from initial attacks, like information gathering attempts or casual threats against known vulnerabilities. In addition, these layers of security should be applied within the environment so that breaking into one server after getting a pivot point in the environment should be just as difficult (if not more difficult) than the original attack that created the pivot point. If “security through obscurity” tactics make up one layer of a multi-layered solution, I’d encourage you to obscure your environment as long as it doesn’t affect your availability.

The key takeaway is:

Security through obscurity is effective if it’s one layer in a multi-layer security solution

Let’s get back to the original purpose of the post.

The biggest benefit to changing the port is to avoid being seen by casual scans. The vast majority of people hunting for any open ssh servers will look for port 22. Some will try the usual variants, like 222 and 2222, but those are few and far between. I ran an experiment with a virtual machine exposed to the internet which had sshd listening on port 22. The server stayed online for one week and then I changed the ssh port to 222. The number of attacks dropped by 98%. Even though this is solely empirical evidence, it’s clear that moving off the standard ssh port reduces your server’s profile.

If it’s more difficult to scan for your ssh server, your chances of being attacked with an ssh server exploit are reduced. A determined attacker can still find the port if they know your server’s IP address via another means (perhaps via a website you host) and they can launch attacks once they find it. Paranoid server administrators might want to check into port knocking to reduce that probability even further.

Remembering the non-standard ssh port can be annoying, but if you have a standard set of workstations that you use for access your servers, just utilize your ~/.ssh/config file to specify certain ports for certain servers. For example:

Host *.mycompany.com
  Port 4321
 
Host nonstandard.mypersonalstuff.com
  Port 2345
 
Host *.mypersonalstuff.com
  Port 5432

If you run into SELinux problems with a non-standard ssh port, there are plenty of guides on this topic.. The setroubleshoot-server package helps out with this as well.

# semanage port -a -t ssh_port_t -p tcp 4321
# semanage port -l | grep ssh
ssh_port_t                     tcp      4321,22

Here is my list of ssh lockdown practices when I build a new server:

  • Update the ssh server package and ensure that automatic updates are configured
  • Enable SELinux and allow a non-standard ssh port
  • Add my ssh public key to the server
  • Disable password logins for ssh
  • Adjust my AllowUsers setting in sshd_config to only allow my user
  • Disable root logins
  • For servers with sensitive data, I install fail2ban
Tagged , , , , , , , , , , ,

Automate CentOS 6 deployments with CIS Security Benchmarks already applied

A coworker heard me grumbling about Linux system administration standards and recommended that I review the CIS Security Benchmarks. After downloading the Red Hat Enterprise Linux 6 security benchmark PDF, I quickly started to see the value of the document. Some of the standards were the installation defaults, some were often forgotten settings, and some were completely brand new to me.

Automating the standards can be a little treacherous simply due to the number of things to adjust and check. I’ve created a kickstart for CentOS 6 and tossed it on Github:

Be sure to read the disclaimers in the README before getting started. Also, keep in mind that the kickstarts are in no way approved by or affiliated with the Center for Internet Security in any way. This is just something I’m offering up to the community in the hope that it helps someone.

Tagged , , , , , , ,

Limit access to the su command

The wheel group exists for a critical purpose and Wikipedia has a concise definition:

In computing, the term wheel refers to a user account with a wheel bit, a system setting that provides additional special system privileges that empower a user to execute restricted commands that ordinary user accounts cannot access. The term is derived from the slang phrase big wheel, referring to a person with great power or influence.

On Red Hat systems (including Fedora), the default sudo configuration allows users in the wheel group to use sudo while all others are restricted from using it in /etc/sudoers:

## Allows people in group wheel to run all commands
%wheel        ALL=(ALL)       ALL

However, the su command can be used by all users by default (which is something I often forget). Fixing it is easy once you take a look at /etc/pam.d/su:

# Uncomment the following line to require a user to be in the "wheel" group.
#auth		required	pam_wheel.so use_uid

Uncomment the line and access to su will only be available for users in the wheel group.

Tagged , , , , , ,