Tag Archives: centos

Migrate KVM virtual machines from CentOS 6 to Fedora 18 without the luxury of shared storage

I’ve converted one of my KVM hypervisors from CentOS 6 to Fedora 18 and now comes the task of migrating my virtual machines off of my single remaining CentOS 6 hypervisor. This is definitely on a budget, so there’s no shared storage to make this process easier.

Here’s how I did it:

Migrate the logical volume
My first VM to migrate is my Fedora development VM where I build and test new packages. I have a 10G logical volume on the old node:

[root@helium ~]# lvs /dev/mapper/vg_helium-fedora--dev 
  LV         VG        Attr     LSize  Pool Origin Data%  Move Log Copy%  Convert
  fedora-dev vg_helium -wi-a--- 10.00g

I made a 10G logical volume on the new hypervisor:

[root@hydrogen ~]# lvcreate -n fedora-dev -L10G vg_hydrogen
  Logical volume "fedora-dev" created

After getting ssh keys set up between both hypervisors and installing pv (to track progress), I started the storage migration over ssh:

dd if=/dev/mapper/vg_helium-fedora--dev | pv | ssh hydrogen dd of=/dev/mapper/vg_hydrogen-fedora--dev

Luckily it was only a 10GB logical volume so it transferred over in a few minutes.

Dump and adjust the source VM’s XML
On the source server, I dumped the VM configuration to an XML file and copied it to the new host:

virsh dumpxml fedora-dev > fedora-dev.xml
scp fedora-dev.xml hydrogen:

Before importing the XML file on the new host, there are some adjustments that need to be made. First off was an adjustment of the storage volume since the new host had the same logical volume name but a different volume group (the source line):

<disk type='block' device='disk'>
  <driver name='qemu' type='raw' cache='none' io='native'></driver>
  <source dev='/dev/vg_hydrogen/fedora-dev'/>
  <target dev='vda' bus='virtio'></target>
  <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'></address>
</disk>

Also, there’s a mismatch with the machine type (not architecture) between CentOS 6 and Fedora 18. I dumped the XML from a VM running on the Fedora 18 hypervisor and compared the machine type to my old CentOS VM’s XML (the XML from the CentOS VM is on top):

-    <type arch='x86_64' machine='rhel6.3.0'>hvm</type>
+    <type arch='x86_64' machine='pc-1.2'>hvm</type>

I replaced rhel6.3.0 with pc-1.2. If you forget this step, your VM won’t start. You’ll get some errors about a mismatched machine type before the VM boots.

There’s one last fix: the path to the qemu-kvm emulator:

-    <emulator>/usr/libexec/qemu-kvm</emulator>
+    <emulator>/usr/bin/qemu-kvm</emulator>

Replace /usr/libexec/qemu-kvm with /usr/bin/qemu-kvm and save your XML file.

Import the VM configuration and launch the VM
Importing the VM on the Fedora 18 hypervisor was easy:

virsh define fedora-dev.xml

That causes the configuration to load into libvirt and it should appear in virt-manager or virsh list by this point. If not, double check your previous steps and look for error messages in your logs. That doesn’t actually start the virtual machine, so I started it on the command line:

virsh start fedora-dev

Within a few moments, the VM was up and responding to pings.

It’s a good idea to hop into virt-manager and verify that the VM configuration is what you expect. Some configuration options don’t line up terribly well between CentOS 6 and Fedora 18. You might need to adjust a few to match the performance you expect to see.

Tagged , , , , , ,

Handling terminal color escape sequences in less

This post is a quick one but I wanted to share it since I taught it to someone new today. When you have bash output with colors, less doesn’t handle the color codes properly by default:

$ colordiff chunk/functions.php chunk-old/functions.php | less
ESC[0;32m22a23,27ESC[0;0m
ESC[0;34m>       * Load up our functions for grabbing content from postsESC[0;0m
ESC[0;34m>       */ESC[0;0m
ESC[0;34m>      require( get_template_directory() . '/content-grabbers.php' );ESC[0;0m
ESC[0;34m> ESC[0;0m

Toss in the -R flag and you’ll be able to see the colors properly (no colors to see here, but use your imagination):

$ colordiff chunk/functions.php chunk-old/functions.php | less -R
22a23,27
>        * Load up our functions for grabbing content from posts
>        */
>       require( get_template_directory() . '/content-grabbers.php' );
> 
>       /**

The man page for less explains the feature in greater detail:

-R or --RAW-CONTROL-CHARS
       Like -r, but only ANSI "color" escape sequences are output in "raw" form.  Unlike -r, the screen appear-
       ance is maintained correctly in most cases.  ANSI "color" escape sequences are sequences of the form:
            ESC [ ... m
       where the "..." is zero or more color specification characters For  the  purpose  of  keeping  track  of
       screen  appearance,  ANSI  color escape sequences are assumed to not move the cursor.  You can make less
       think that characters other than "m" can end ANSI color escape  sequences  by  setting  the  environment
       variable  LESSANSIENDCHARS to the list of characters which can end a color escape sequence.  And you can
       make less think that characters other than the standard ones may appear between the ESC  and  the  m  by
       setting the environment variable LESSANSIMIDCHARS to the list of characters which can appear.
Tagged , , , , , ,

Automate CentOS 6 deployments with CIS Security Benchmarks already applied

A coworker heard me grumbling about Linux system administration standards and recommended that I review the CIS Security Benchmarks. After downloading the Red Hat Enterprise Linux 6 security benchmark PDF, I quickly started to see the value of the document. Some of the standards were the installation defaults, some were often forgotten settings, and some were completely brand new to me.

Automating the standards can be a little treacherous simply due to the number of things to adjust and check. I’ve created a kickstart for CentOS 6 and tossed it on Github:

Be sure to read the disclaimers in the README before getting started. Also, keep in mind that the kickstarts are in no way approved by or affiliated with the Center for Internet Security in any way. This is just something I’m offering up to the community in the hope that it helps someone.

Tagged , , , , , , ,

Limit access to the su command

The wheel group exists for a critical purpose and Wikipedia has a concise definition:

In computing, the term wheel refers to a user account with a wheel bit, a system setting that provides additional special system privileges that empower a user to execute restricted commands that ordinary user accounts cannot access. The term is derived from the slang phrase big wheel, referring to a person with great power or influence.

On Red Hat systems (including Fedora), the default sudo configuration allows users in the wheel group to use sudo while all others are restricted from using it in /etc/sudoers:

## Allows people in group wheel to run all commands
%wheel        ALL=(ALL)       ALL

However, the su command can be used by all users by default (which is something I often forget). Fixing it is easy once you take a look at /etc/pam.d/su:

# Uncomment the following line to require a user to be in the "wheel" group.
#auth		required	pam_wheel.so use_uid

Uncomment the line and access to su will only be available for users in the wheel group.

Tagged , , , , , ,

Reprint: Stop Disabling SELinux!

This article appeared in SC Magazine and I’ve posted it here as well. For those of you who were left wanting more from my previous SELinux post, this should help. If it doesn’t help, leave a comment. ;)


The push to cloud transforms the way we apply information security principles to systems and applications. Perimeters of the past, secured heavily with traditional network devices in the outermost ring, lose effectiveness day by day. Shifting the focus to “defense in depth” brings the perimeter down to the individual cloud instances running your application. Security-Enhanced Linux, or SELinux, forms an effective part of that perimeter.

SELinux operates in the realm of mandatory access control, or MAC. The design of MAC involves placing constraints on what a user (a subject) can do to a particular object (a target) on the system. In contrast, discretionary access control, or DAC, allows a user with certain access to use discretion to limit or allow access to certain files, directories, or devices. You can set any file system permissions that you want but SELinux can override them with ease at the operating system level.

Consider a typical server running a web application. An attacker compromises the web application and executes malicious code via the web server daemon itself. SELinux has default policies that prevent the daemon from initiating communication on the network. That limits the attacker’s options to attack other services or servers.

In addition, SELinux sets policies on which files and directories the web server can access, regardless of any file system permissions. This protection limits the attacker’s access to other sensitive parts of the file system even if the administrator set the files to be readable to the world.

This is where SELinux shines. Oddly enough, this is the point where many system administrators actually disable SELinux on their systems.

Troubleshooting these events, called AVC denials, without some helpful tools is challenging and frustrating. Each denial flows into to your audit log as a cryptic message. Most administrators will check the usual suspects, like firewall rules and file system permissions. As frustration builds, they disable SELinux and notice that their application begins working as expected. SELinux remains disabled and hundreds of helpful policies lie dormant solely because one policy caused a problem.

Disabling SELinux without investigation frustrated me to the point where I started a site at stopdisablingselinux.com. The site is a snarky response to Linux administrators who reach for the disable switch as soon as SELinux gets in their way.

All jokes aside, here are some helpful tips to use SELinux effectively:

Use the setroubleshoot helpers to understand denials
Working through denials is easy with the setroubleshoot-server package. When a denial occurs, you still receive a cryptic log message in your audit logs. However, you also receive a message via syslog that is very easy to read. Your server can email you these messages as well. The message contains guidance about adjusting SELinux booleans, setting contexts, or generating new SELinux policies to work around a really unusual problem. When I say guidance, I mean that the tools give you commands to copy and paste to adjust your policies, booleans and contexts.

Review SELinux booleans for quick adjustments
Although the myriad of SELinux user-space tools isn’t within the scope of this article, getsebool and togglesebool deserve a mention. Frequently adjusted policies are controlled by booleans that are toggled on and off with togglesebool. Start with getsebool –a for a full list of booleans and then use togglesebool to enable or disable the policy.

Quickly restore file or directory contexts
Shuffling files or directories around a server can cause SELinux denials due to contexts not matching their original values. This happens to me frequently if I move a configuration file from one system to another. Correcting the context problem involves one of two simple commands. The restorecon command applies the default contexts specific to the file or directory. If you have a file in the directory with the correct context, use chcon to fix the context on the wrong file by giving it the path to the file with the correct context.

Here are some additional links with helpful SELinux documentation:

Tagged , , , , , ,