Virtualization on Major Hayden

OpenStack isn’t dead. It’s boring. That’s a good thing.

major@mhtx.net (Major Hayden) — Fri, 24 Feb 2017 16:06:24 +0000

NOTE: The opinions shared in this post are mine alone and are not related to my employer in any way.

The first OpenStack Project Teams Gathering (PTG) event was held this week in Atlanta. The week was broken into two parts: cross-project work on Monday and Tuesday, and individual projects Wednesday through Friday. I was there for the first two days and heard a few discussions that started the same way.

Everyone keeps saying OpenStack is dead.

Is it?

OpenStack isn’t dead. It’s boring.

“The report of my death was an exaggeration”

Mark Twain said it best, but it works for OpenStack as well. The news has plenty of negative reports that cast a shadow over OpenStack’s future. You don’t have to look far to find them:

This isn’t evidence of OpenStack’s demise, but rather a transformation. Gartner called OpenStack a “science project” in 2015 and now 451 Research Group is saying something very different:

451 Research Group estimates OpenStack’s ecosystem to grow nearly five-fold in revenue, from US$1.27 billion market size in 2015 to US$5.75 billion by 2020.

A 35% CAGR sounds pretty good for a product in the middle of a transformation. In Texas, we’d say that’s more than enough to “shake a stick at”.

The transformation

You can learn a lot about the transformation going on within OpenStack by reading analyst reports and other news online. I won’t go into that here since that data is readily available.

Instead, I want to take a look at how OpenStack has changed from the perspective of a developer. My involvement with OpenStack started in the Diablo release in 2011 and my first OpenStack Summit was the Folsom summit in San Francisco.

Much of the discussion at that time was around the “minutiae” of developing software in its early forms. We discussed topics like how to test, how to handle a myriad of requirements that constantly change, and which frameworks to use in which projects. The list of projects was quite short at that time (there were only 7 main services in Grizzly). Lots of effort certainly poured into feature development, but there was a ton of work being done to keep the wheels from falling off entirely.

The discussions at this week’s PTG were very different.

Most of the discussion was around adding new integrations, improving reliability, and increasing scale. Questions were asked about how to integrate OpenStack into existing enterprise processes and applications. Reliability discussions were centered less around making the OpenStack services reliable, but more around how to increase overall resiliency when other hardware or software is misbehaving.

Discussions or arguments about minutiae were difficult to find.

Boring is good

I’m not trying to say that working with OpenStack is boring. Developing software within the OpenStack community is an enjoyable experience. The rules and regulations within most projects are there to prevent design mistakes that have appeared before and many of these sets of rules are aligned between projects. Testing code and providing meaningful reviews is also straightforward.

However, the drama, both unproductive and productive, that plagued the project in the past is diminishing. It still exists in places, especially when it comes to vendor relationships. (That’s where most open source projects see their largest amounts of friction, anyway.)

This transformation may make OpenStack appear “dead” to some. The OpenStack community is solving different problems now. Many of them are larger and more difficult to solve. Sometimes these challenges take more than one release to overcome. Either way, many OpenStack developers are up for these new challenges, even if they don’t make the headlines.

As for me: bring on the boring. Let’s crush the hard stuff.

Photo credit: By Mike (Flickr: DSC_6831_2_3_tonemapped) [CC BY 2.0], via Wikimedia Commons

OpenStack instances come online with multiple network ports attached

major@mhtx.net (Major Hayden) — Wed, 03 Aug 2016 14:40:16 +0000

I ran into an interesting problem recently in my production OpenStack deployment that runs the Mitaka release. On various occasions, instances were coming online with multiple network ports attached, even though I only asked for one network port.

The problem

If I issued a build request for ten instances, I’d usually end up with this:

6 instances with one network port attached
2-3 instances with two network ports attached (not what I want)
1-2 instances with three or four network ports attached (definitely not what I want)

When I examined the instances with multiple network ports attached, I found that one of the network ports would be marked as up while the others would be marked as down. However, the IP addresses associated with those extra ports would still be associated with the instance in horizon and via the nova API. All of the network ports seemed to be fully configured on the neutron side.

Digging into neutron

The neutron API logs are fairly chatty, especially while instances are building, but I found two interesting log lines for one of my instances:

172.29.236.41,172.29.236.21 - - [02/Aug/2016 14:03:11] "GET /v2.0/ports.json?tenant_id=a7b0519330ed481884431102a72dd04c&device_id=05eef1bb-5356-43d9-86c9-4d9854d4d46b HTTP/1.1" 200 2137 0.025282
172.29.236.11,172.29.236.21 - - [02/Aug/2016 14:03:15] "GET /v2.0/ports.json?tenant_id=a7b0519330ed481884431102a72dd04c&device_id=05eef1bb-5356-43d9-86c9-4d9854d4d46b HTTP/1.1" 200 3098 0.027803

There are two requests to create network ports for this instance and neutron is allocating ports to both requests. This would normally be just fine, but I only asked for one network port on this instance.

The IP addresses making the requests are unusual, though. 172.29.236.11 and 172.29.236.41 are two of the hypervisors within my cloud. Why are both of them asking neutron for network ports? Only one of those hypervisors should be building my instance, not both. After checking both hypervisors, I verified that the instance was only provisioned on one of the hosts and not both.

Looking at nova-compute

The instance ended up on the 172.29.236.11 hypervisor once it finished building and the logs on that hypervisor looked fine:

nova.virt.libvirt.driver [-] [instance: 05eef1bb-5356-43d9-86c9-4d9854d4d46b] Instance spawned successfully.

I logged into the 172.29.236.41 hypervisor since it was the one that asked neutron for a port but it never built the instance. The logs there had a much different story:

[instance: 05eef1bb-5356-43d9-86c9-4d9854d4d46b] Instance failed to spawn
Traceback (most recent call last):
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/compute/manager.py", line 2218, in _build_resources
 yield resources
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/compute/manager.py", line 2064, in _build_and_run_instance
 block_device_info=block_device_info)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 2773, in spawn
 admin_pass=admin_password)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 3191, in _create_image
 instance, size, fallback_from_host)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/virt/libvirt/driver.py", line 6765, in _try_fetch_image_cache
 size=size)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 251, in cache
 *args, **kwargs)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 591, in create_image
 prepare_template(target=base, max_size=size, *args, **kwargs)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 271, in inner
 return f(*args, **kwargs)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/virt/libvirt/imagebackend.py", line 241, in fetch_func_sync
 fetch_func(target=target, *args, **kwargs)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/virt/libvirt/utils.py", line 429, in fetch_image
 max_size=max_size)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/virt/images.py", line 120, in fetch_to_raw
 max_size=max_size)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/virt/images.py", line 110, in fetch
 IMAGE_API.download(context, image_href, dest_path=path)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/image/api.py", line 182, in download
 dst_path=dest_path)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/image/glance.py", line 383, in download
 _reraise_translated_image_exception(image_id)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/image/glance.py", line 682, in _reraise_translated_image_exception
 six.reraise(new_exc, None, exc_trace)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/image/glance.py", line 381, in download
 image_chunks = self._client.call(context, 1, 'data', image_id)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/nova/image/glance.py", line 250, in call
 result = getattr(client.images, method)(*args, **kwargs)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/glanceclient/v1/images.py", line 148, in data
 % urlparse.quote(str(image_id)))
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/glanceclient/common/http.py", line 275, in get
 return self._request('GET', url, **kwargs)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/glanceclient/common/http.py", line 267, in _request
 resp, body_iter = self._handle_response(resp)
 File "/openstack/venvs/nova-13.3.0/lib/python2.7/site-packages/glanceclient/common/http.py", line 83, in _handle_response
 raise exc.from_response(resp, resp.content)
ImageNotFound: Image 8feacda9-91fd-48ce-b983-54f7b6de6650 could not be found.

This is one of those occasions where I was glad to find an exception in the log. The image that couldn’t be found is an image I’ve used regularly in the environment before, and I know it exists.

Gandering at glance

First off, I asked glance what it knew about the image:

$ openstack image show 8feacda9-91fd-48ce-b983-54f7b6de6650
+------------------+------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------+
| checksum | 8de08e3fe24ee788e50a6a508235aa64 |
| container_format | bare |
| created_at | 2016-08-03T01:25:34Z |
| disk_format | qcow2 |
| file | /v2/images/8feacda9-91fd-48ce-b983-54f7b6de6650/file |
| id | 8feacda9-91fd-48ce-b983-54f7b6de6650 |
| min_disk | 0 |
| min_ram | 0 |
| name | Fedora 24 |
| owner | a7b0519330ed481884431102a72dd04c |
| properties | description='' |
| protected | False |
| schema | /v2/schemas/image |
| size | 204590080 |
| status | active |
| tags | |
| updated_at | 2016-08-03T01:25:39Z |
| virtual_size | None |
| visibility | public |
+------------------+------------------------------------------------------+

If glance knows about the image, why can’t that hypervisor build an instance with that image? While I was scratching my head, Kevin Carter walked by my desk and joined in the debugging.

He asked about how I had deployed glance and what storage backend I was using. I was using the regular file storage backend since I don’t have swift deployed in the environment. He asked me how many glance nodes I had (I had two) and if I was doing anything to sync the images between the glance nodes.

Then it hit me.

Although both glance nodes knew about the image (since that data is in the database), only one of the glance nodes had the actual image content (the actual qcow2 file) stored. That means that if a hypervisor requests the image from a glance node that knows about the image but doesn’t have it stored, the hypervisor won’t be able to retrieve the image.

Unfortunately, the checks go in this order on the nova-compute side:

Ask glance if this image exists and if this tenant can use it
Configure the network
Retrieve the image

If a hypervisor rolls through steps one and two without issues, but then fails on step 3, the network port will be provisioned but won’t come up on the instance. There’s nothing that cleans up that port in the Mitaka release, so it requires manual intervention.

The fix

As a temporary workaround, I took one of the glance nodes offline so that only one glance node is being used. After hundreds of builds, all of the instances came up with only one network port attached!

There are a few options for long-term fixes.

I could deploy swift and put glance images into swift. That would allow me to use multiple glance nodes with the same swift backend. Another option would be to use an existing swift deployment, such as Rackspace’s Cloud Files product.

Since I’m not eager to deploy swift in my environment for now, I decided to remove the second glance node and reconfigure nova to use only one glance node. That means I’m running with only one glance node and a failure there could be highly annoying. However, that trade-off is fine with me until I can get around to deploying swift.

UPDATE: I’ve opened a bug for nova so that the network ports are cleaned up if the instance fails to build.

Photo credit: Flickr: pascalcharest

Talk Recap: Automated security hardening with OpenStack-Ansible

major@mhtx.net (Major Hayden) — Tue, 26 Apr 2016 21:19:02 +0000

Today is the second day of the OpenStack Summit in Austin and I offered up a talk on host security hardening in OpenStack clouds. You can download the slides or watch the video here:

Here’s a quick recap of the talk and the conversations afterward:

Security tug-of-war

Information security is a challenging task, mainly because it is more than just a technical problem. Technology is a big part of it, but communication, culture, and compromise are also critical. I flashed up this statement on the slides:

"People should feel like security is something they are part of; not something that is done to them" @majorhayden pic.twitter.com/Blh9rZp0uL

— Rackspace (@Rackspace) April 26, 2016

In the end, the information security teams, the developers and the auditors must be happy. This can be a challenging tightrope to walk, but automating some security allows everyone to get what they want in a scalable and repeatable way.

Meeting halfway

The openstack-ansible-security role allows information security teams to meet developers or OpenStack deployers halfway. It can easily bolt onto existing Ansible playbooks and manage host security hardening for Ubuntu 14.04 systems. The role also works in non-OpenStack environments just as well. All of the documentation, configuration, and Ansible tasks are all included with the role.

The role itself applies security configurations to each host in an environment. Those configurations are based on the Security Technical Implementation Guide (STIG) from the Defense Information Systems Agency (DISA), which is part of the United States Department of Defense. The role takes the configurations from the STIG and makes small tweaks to fit an OpenStack environment. All of the tasks are carefully translated from the STIG for Red Hat Enterprise Linux 6 (there is no STIG for Ubuntu currently).

The role is available now as part of OpenStack-Ansible in the Liberty, Mitaka, and Newton releases. Simply adjust apply_security_hardening from false to true and deploy. For other users, the role can easily be used in any Ansible playbook. (Be sure to review the configuration to ensure its defaults meet your requirements.)

Getting involved

We need your help! Upcoming plans include Ubuntu 16.04 and CentOS support, a rebase onto the RHEL 7 STIG (which will be finalized soon), and better reporting.

Join us later this week for the OpenStack-Ansible design summit sessions or anytime on Freenode in #openstack-ansible. We’re on the OpenStack development mailing list as well (be sure to use the [openstack-ansible][security] tags.

Hallway conversations

Lots of people came by to chat afterwards and offered to join in the development. A few people were hoping it would have been the security “silver bullet”, and I reset some expectations.

Some attendees has good ideas around making the role more generic and adding an “OpenStack switch” that would configure many variables to fit an OpenStack environment. That would allow people to use it easily with non-OpenStack environments.

Other comments were around hardening inside of Linux containers. These users had “heavy” containers where the entire OS is virtualized and multiple processes might be running at the same time. Some of the configuration changes (especially the kernel tunables) don’t make sense inside a container like that, but many of the others could be useful. For more information on securing Linux containers, watch the video from Thomas Cameron’s talk here at the summit.

Thank you

I’d like to thank everyone for coming to the talk today and sharing their feedback. It’s immensely useful and I pile all of that feedback into future talks. Also, I’d like to thank all of the people at Rackspace who helped me review the slides and improve them.

systemd-networkd and macvlan interfaces

major@mhtx.net (Major Hayden) — Mon, 26 Oct 2015 13:50:36 +0000

I spent some time working with macvlan interfaces on KVM hypervisors last weekend. They’re interesting because they’re not really a bridge. It allows you to assign multiple MAC addresses to a single interface and then allow the kernel to filter traffic into tap interfaces based on the MAC address in the packet. If you’re looking for a highly detailed explanation, head on over to waldner’s blog for a deep dive into the technology and the changes that come along with it.

Why macvlan?

Bridging can become a pain to work with, especially when you’re forced to add in creative filtering rules and keep them updated. The macvlan interfaces can help with that (read up on VEPA mode). There are some interesting email threads showing that macvlan interfaces can improve network performance for various workloads. Low latency workloads can benefit from the simplicity and low overhead of macvlan interfaces.

systemd-networkd and macvlan interfaces

Fortunately for us, systemd-networkd makes configuring a macvlan interface really easy. I’ve written about configuring bridges with systemd-networkd and the process for macvlan interfaces is similar.

In my scenario, I have a 1U server with an ethernet interface called enp4s0 (read up on interface naming with systemd-udevd). I want to make a macvlan interface for virtual machines and I’ll be attaching VM’s to that interface via macvtap interfaces. It’s similar to bridging where you make a bridge and then give everyone a port on the bridge.

Start by creating a network device for our macvlan interface:

# /etc/systemd/network/vmbridge.netdev
[NetDev]
Name=vmbridge
Kind=macvlan

[MACVLAN]
Mode=bridge

I’ve told systemd-networkd that I want a macvlan interface set up in bridge mode. This will allow hosts and virtual machines to talk to one another on the interface. You could choose vepa for the mode if you want additional security. However, this will force traffic out to your upstream switch/router and makes it challenging for hosts and guests to communicate with each other.

Now that we have a device configured, let’s configure the IP address for the macvlan interface (similar to configuring a bridge):

# /etc/systemd/network/vmbridge.network
[Match]
Name=vmbridge

[Network]
IPForward=yes
Address=192.168.250.33/24
Gateway=192.168.250.1
DNS=192.168.250.1

Let’s tell systemd-networkd that our physical network interface, enp4s0, is part of this interface:

# /etc/systemd/network/enp4s0.network
[Match]
Name=enp4s0

[Network]
MACVLAN=vmbridge

This is very similar to a configuration for a standard Linux bridge. Once you’ve reached this step, you’ll most likely want to reboot to ensure all of your network devices come up properly.

Attaching a virtual machine

Attaching a KVM virtual machine to the macvlan interface is quite easy. When you’re creating a new VM using virt-manager, look for this setting in the wizard:

If you’re installing via virt-install just use the following argument for your network configuration:

--network type=direct,source=vmbridge,source_mode=bridge

You’ll end up with interfaces like these after creating multiple virtual machines:

 mtu 1500 qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 500
 link/ether 52:54:00:83:53:f2 brd ff:ff:ff:ff:ff:ff promiscuity 0
 macvtap mode bridge addrgenmode eui64
15: macvtap2@enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 500
 link/ether 52:54:00:f1:76:0b brd ff:ff:ff:ff:ff:ff promiscuity 0
 macvtap mode bridge addrgenmode eui64
17: macvtap3@enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 500
 link/ether 52:54:00💿53:34 brd ff:ff:ff:ff:ff:ff promiscuity 0
 macvtap mode bridge addrgenmode eui64
20: macvtap1@enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN mode DEFAULT group default qlen 500
 link/ether 52:54:00:18:79:d3 brd ff:ff:ff:ff:ff:ff promiscuity 0
 macvtap mode bridge addrgenmode eui64

Live migration failures with KVM and libvirt

major@mhtx.net (Major Hayden) — Mon, 03 Aug 2015 13:13:30 +0000

I decided to change some of my infrastructure back to KVM again, and the overall experience has been quite good in Fedora 22. Using libvirt with KVM is a breeze and the virt-manager tools make it even easier. However, I ran into some problems while trying to migrate virtual machines from one server to another.

The error

# virsh migrate --live --copy-storage-all bastion qemu+ssh://root@192.168.250.33/system
error: internal error: unable to execute QEMU command 'drive-mirror': Failed to connect socket: Connection timed out

That error message wasn’t terribly helpful. I started running through my usual list of checks:

Can the hypervisors talk to each other? Yes, iptables is disabled.
Are ssh keys configured? Yes, verified.
What about ssh host keys being accepted on each side? Both sides can ssh without interaction.
SELinux? No AVC’s logged.
Libvirt logs? Nothing relevant in libvirt’s qemu logs.
Filesystem permissions for libvirt’s directories? Identical on both sides.
Libvirt daemon running on both sides? Yes.

I was pretty confused at this point. A quick Google search didn’t reveal too many relevant issues, but I did find a Red Hat Bug from 2013 that affected RHEL 7. The issue in the bug was that libvirt wasn’t using the right ports to talk between servers and those packets were being dropped by iptables. My iptables rules were empty.

Debug time

I ran the same command with LIBVIRT_DEBUG=1 at the front:

 debug.log

After scouring the pages and pages of output, I couldn’t find anything useful.

Eureka!

I spotted an error message briefly in virt-manager or the debug logs that jogged my brain to think about a potential problem: hostnames. Both hosts had a fairly bare /etc/hosts file without IP/hostname pairs for each hypervisor. After editing both servers’ /etc/hosts file to include the short and full hostnames for each hypervisor, I tested the live migration one more time.

Success!

The migration went off without a hitch in virt-manager and via the virsh client. I migrated several VM’s, including the one running this site, with no noticeable interruption.

Try out LXC with an Ansible playbook

major@mhtx.net (Major Hayden) — Wed, 17 Dec 2014 13:50:26 +0000

The world of containers is constantly evolving lately. The latest turn of events involves the CoreOS developers when they announced Rocket as an alternative to Docker. However, LXC still lingers as a very simple path to begin using containers.

When I talk to people about LXC, I often hear people talk about how difficult it is to get started with LXC. After all, Docker provides an easy-to-use image downloading function that allows you to spin up multiple different operating systems in Docker containers within a few minutes. It also comes with a daemon to help you manage your images and your containers.

Managing LXC containers using the basic LXC tools isn’t terribly easy - I’ll give you that. However, managing LXC through libvirt makes the process much easier. I wrote a little about this earlier in the year.

I decided to turn the LXC container deployment process into an Ansible playbook that you can use to automatically spawn an LXC container on any server or virtual machine. At the moment, only Fedora 20 and 21 are supported. I plan to add CentOS 7 and Debian support soon.

Clone the repository to get started:

git clone https://github.com/major/ansible-lxc.git
cd ansible-lxc
ansible-playbook -i hosts playbook.yml

If you’re running the playbook on the actual server or virtual machine where you want to run LXC, there’s no need to alter the hosts file. You will need to adjust it if you’re running your playbook from a remote machine.

As the playbook runs, it will install all of the necessary packages and begin assembling a Fedora 21 chroot. It will register the container with libvirt and do some basic configuration of the chroot so that it will work as a container. You’ll end up with a running Fedora 21 LXC container that is using the built-in default NAT network created by libvirt. The playbook will print out the IP address of the container at the end. The default password for root is fedora. I wouldn’t recommend leaving that for a production use container. ;)

All of the normal virsh commands should work on the container. For example:

# Stop the container gracefully
virsh shutdown fedora21
# Start the container
virsh start fedora21

Feel free to install the virt-manager tool and manage everything via a GUI locally or via X forwarding:

yum -y install virt-manager dejavu* xorg-x11-xauth
# OPTIONAL: For a better looking virt-manager interface, install these, too
yum -y install gnome-icon-theme gnome-themes-standard

Configure static IP addresses for Project Atomic’s KVM image

major@mhtx.net (Major Hayden) — Wed, 23 Apr 2014 15:14:39 +0000

Amid all of the Docker buzz at the Red Hat Summit, Project Atomic was launched. It’s a minimalistic Fedora 20 image with a few tweaks, including rpm-ostree and geard.

There are great instructions on the site for firing up a test instance under KVM but my test server doesn’t have a DHCP server on its network. You can use Project Atomic with static IP addresses fairly easily:

Create a one-line /etc/sysconfig/network:

NETWORKING=yes

Drop in a basic network configuration into /etc/sysconfig/network-scripts/ifcfg-eth0:

DEVICE=eth0
IPADDR=10.127.92.32
NETMASK=255.255.255.0
GATEWAY=10.127.92.1
ONBOOT=yes

All that’s left is to set DNS servers and a hostname:

echo "nameserver 8.8.8.8" > /etc/resolv.conf
hostnamectl set-hostname myatomichost.example.com

Bring up the network interface:

ifup eth0

Of course, you could do all of this via the nmcli tool if you prefer to go that route.

Launch secure LXC containers on Fedora 20 using SELinux and sVirt

major@mhtx.net (Major Hayden) — Tue, 22 Apr 2014 04:11:00 +0000

Getting started with LXC is a bit awkward and I’ve assembled this guide for anyone who wants to begin experimenting with LXC containers in Fedora 20. As an added benefit, you can follow almost every step shown here when creating LXC containers on Red Hat Enterprise Linux 7 Beta (which is based on Fedora 19).

You’ll need a physical machine or a VM running Fedora 20 to get started. (You could put a container in a container, but things get a little dicey with that setup. Let’s just avoid talking about nested containers for now. No, really, I shouldn’t have even brought it up. Sorry about that.)

Prep Work

Start by updating all packages to the latest versions available:

yum -y upgrade

Verify that SELinux is in enforcing mode by running getenforce. If you see Disabled or Permissive, get SELinux into enforcing mode with a quick configuration change:

sed -i 's/^SELINUX=.*/SELINUX=enforcing/' /etc/selinux/config

I recommend installing setroubleshoot-server to make it easier to find the root cause of AVC denials:

Reboot now. This will ensure that SELinux comes up in enforcing mode (verify that with getenforce after reboot) and it ensures that auditd starts up sedispatch (for setroubleshoot).

Install management libraries and utilities

Let’s grab libvirt along with LXC support and a basic NAT networking configuration.

yum -y install libvirt-daemon-lxc libvirt-daemon-config-network

Launch libvirtd via systemd and ensure that it always comes up on boot. This step will also adjust firewalld for your containers and ensure that dnsmasq is serving up IP addresses via DHCP on your default NAT network.

systemctl start libvirtd.service
systemctl enable libvirtd.service

Bootstrap our container

Installing packages into the container’s filesystem will take some time.

yum -y --installroot=/var/lib/libvirt/filesystems/fedora20 --releasever=20 --nogpg install systemd passwd yum fedora-release vim-minimal openssh-server procps-ng iproute net-tools dhclient

This step fills in the filesystem with the necessary packages to run a Fedora 20 container. We now need to tell libvirt about the container we’ve just created.

virt-install --connect lxc:// --name fedora20 --ram 512 --filesystem /var/lib/libvirt/filesystems/fedora20/,/

At this point, libvirt will know enough about the container to start it and you’ll be connected to the console of the container! We need to adjust some configuration files within the container to use it properly. Detach from the console with CTRL-].

Let’s stop the container so we can make some adjustments.

virsh -c lxc:// shutdown fedora20

Get the container ready for production

Hop into your container and set a root password.

chroot /var/lib/libvirt/filesystems/fedora20 /bin/passwd root

We will be logging in as root via the console occasionally and we need to allow that access.

echo "pts/0" >> /var/lib/libvirt/filesystems/fedora20/etc/securetty

Since we will be using our NAT network with our auto-configured dnsmasq server (thanks to libvirt), we can configure a simple DHCP setup for eth0:

cat < < EOF > /var/lib/libvirt/filesystems/fedora20/etc/sysconfig/network
NETWORKING=yes
EOF
cat < < EOF > /var/lib/libvirt/filesystems/fedora20/etc/sysconfig/network-scripts/ifcfg-eth0
BOOTPROTO=dhcp
ONBOOT=yes
DEVICE=eth0
EOF

Using ssh makes the container a lot easier to manage, so let’s ensure that it starts when the container boots. (You could do this via systemctl after logging in at the console, but I’m lazy.)

chroot /var/lib/libvirt/filesystems/fedora20/
ln -s /usr/lib/systemd/system/sshd.service /etc/systemd/system/multi-user.target.wants/
exit

Launch!

Cross your fingers and launch the container.

virsh -c lxc:// start --console fedora20

You’ll be attached to the console during boot but don’t worry, hold down CTRL-] to get back to your host prompt. Check the dnsmasq leases to find your container’s IP address and you can login as root over ssh.

cat /var/lib/libvirt/dnsmasq/default.leases

Security

After logging into your container via ssh, check the process labels within the container:

# ps aufxZ
LABEL USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 1 0.0 1.3 47444 3444 ? Ss 03:18 0:00 /sbin/init
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 18 0.0 2.0 43016 5368 ? Ss 03:18 0:00 /usr/lib/systemd/systemd-journald
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 38 0.4 7.8 223456 20680 ? Ssl 03:18 0:00 /usr/bin/python -Es /usr/sbin/firewalld -
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 40 0.0 0.7 26504 2084 ? Ss 03:18 0:00 /usr/sbin/smartd -n -q never
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 41 0.0 0.4 19268 1252 ? Ss 03:18 0:00 /usr/sbin/irqbalance --foreground
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 44 0.0 0.6 34696 1636 ? Ss 03:18 0:00 /usr/lib/systemd/systemd-logind
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 46 0.0 1.8 267500 4832 ? Ssl 03:18 0:00 /sbin/rsyslogd -n
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 dbus 47 0.0 0.6 26708 1680 ? Ss 03:18 0:00 /bin/dbus-daemon --system --address=syste
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 rpc 54 0.0 0.5 41992 1344 ? Ss 03:18 0:00 /sbin/rpcbind -w
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 55 0.0 0.3 25936 924 ? Ss 03:18 0:00 /usr/sbin/atd -f
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 56 0.0 0.5 22728 1488 ? Ss 03:18 0:00 /usr/sbin/crond -n
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 60 0.0 0.2 6412 784 pts/0 Ss+ 03:18 0:00 /sbin/agetty --noclear -s console 115200
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 74 0.0 3.2 339808 8456 ? Ssl 03:18 0:00 /usr/sbin/NetworkManager --no-daemon
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 394 0.0 5.9 102356 15708 ? S 03:18 0:00 \_ /sbin/dhclient -d -sf /usr/libexec/nm
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 polkitd 83 0.0 4.4 514792 11548 ? Ssl 03:18 0:00 /usr/lib/polkit-1/polkitd --no-debug
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 rpcuser 110 0.0 0.6 46564 1824 ? Ss 03:18 0:00 /sbin/rpc.statd
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 111 0.0 1.3 82980 3620 ? Ss 03:18 0:00 /usr/sbin/sshd -D
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 409 0.0 1.9 131576 5084 ? Ss 03:18 0:00 \_ sshd: root@pts/1
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 413 0.0 0.9 115872 2592 pts/1 Ss 03:18 0:00 \_ -bash
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 438 0.0 0.5 123352 1344 pts/1 R+ 03:19 0:00 \_ ps aufxZ
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 411 0.0 0.8 44376 2252 ? Ss 03:18 0:00 /usr/lib/systemd/systemd --user
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 412 0.0 0.5 66828 1328 ? S 03:18 0:00 \_ (sd-pam)
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 436 0.0 0.4 21980 1144 ? Ss 03:19 0:00 /usr/lib/systemd/systemd-hostnamed

You’ll notice something interesting if you run getenforce now within the container — SELinux is disabled. Actually, it’s not really disabled. The processing of SELinux policy is done on the host. The container isn’t able to see what’s going on outside of its own files and processes. The libvirt documentation for LXC hints at the importance of this isolation:

A suitably configured UID/GID mapping is a pre-requisite to making containers secure, in the absence of sVirt confinement.

In the absence of the “user” namespace being used, containers cannot be considered secure against exploits of the host OS. The sVirt SELinux driver provides a way to secure containers even when the “user” namespace is not used. The cost is that writing a policy to allow execution of arbitrary OS is not practical. The SELinux sVirt policy is typically tailored to work with an simpler application confinement use case, as provided by the “libvirt-sandbox” project.

This leads to something really critical to understand:

Containers don’t contain

Dan Walsh has a great post that goes into the need for sVirt and the protections it can provide when you need to be insulated from potentially dangerous virtual machines or containers. If a user is root inside a container, they’re root on the host as well. (There’s an exception: UID namespaces. But let’s not talk about that now. Oh great, first it was nested containers and now I brought up UID namespaces. Sorry again.)

Dan’s talk about securing containers hasn’t popped up on the Red Hat Summit presentations page quite yet but here are some notes that I took and then highlighted:

Containers don’t contain. The kernel doesn’t know about containers. Containers simply use kernel subsystems to carve up namespaces for applications.
Containers on Linux aren’t complete. Don’t compare directly to Solaris zones yet.
Running containers without Mandatory Access Control (MAC) systems like SELinux or AppArmor opens the door for full system compromise via untrusted applications and users within containers.

Using MAC gives you one extra barrier to keep a malicious container from getting higher levels of access to the underlying host. There’s always a chance that a kernel exploit could bypass MAC but it certainly raises the level of difficulty for an attacker and allows server operators extra time to react to alerts.

Docker, trusted builds, and Fedora 20

major@mhtx.net (Major Hayden) — Wed, 26 Mar 2014 05:17:58 +0000

Docker is a hot topic in the Linux world at the moment and I decided to try out the new trusted build process. Long story short, you put your Dockerfile along with any additional content into your GitHub repository, link your GitHub account with Docker, and then fire off a build. The Docker index labels it as “trusted” since it was build from source files in your repository.

I set off to build a Dockerfile to provision a container that would run all of the icanhazip services. Getting httpd running was a little tricky, but I soon had a working Dockerfile that built and ran successfully on Fedora 20.

The trusted build process kicked off without much fuss and I found myself waiting for a couple of hours for my job to start. I was sad to see an error after waiting so long:

Installing : httpd-2.4.7-3.fc20.x86_64
error: unpacking of archive failed on file /usr/sbin/suexec: cpio: cap_set_file

Well, that’s weird. It turns out that cap_set_file is part of libcap that sets filesystem capabilities based on the POSIX.1e standards. You can read up on capabilities in the Linux kernel capabilities FAQ. (Special thanks to Andrew Clayton getting me pointed in the right direction there.)

Marek Goldmann ran into this problem back in September 2013 and opened a bug report. Marek proposed a change to the Docker codebase that would remove setfcap from the list of banned capabilities in the LXC template used by docker. Another workaround would be to use the -privileged option to perform a build in privileged mode (available in docker 0.6+).

Both of those workarounds are unavailable when doing trusted builds with docker’s index. Sigh.

I fired off an email to Docker’s support staff and received a quick reply:

Major,

We are aware of this issue, and we are currently working on a fix, and we hope to have something we can start testing this week. I’m not sure when we will be able to roll out the fix, but we are hoping soon. Until then, there isn’t anything you can do to work around it. Sorry for the inconvenience.

If anything changes, we will be sure to let you know.

Ken

It wasn’t the answer I wanted but it’s good to know that the issue is being worked. In the meantime, I’ll push an untrusted build of the icanhazip Docker container up to the index for everyone to enjoy.

Stay tuned for updates.

UPDATED 2014-08-08: Per Thomas’ comment below, this has been fixed upstream.

virt-manager: ‘NoneType’ object has no attribute ‘cpus’

major@mhtx.net (Major Hayden) — Thu, 06 Mar 2014 18:44:58 +0000

After upgrading my Fedora 20 Xen hypervisor to virt-manager 1.0.0, I noticed that I couldn’t open the console or VM details for any of my guests. Running virt-manager --debug gave me the following traceback:

Traceback (most recent call last):
 File "/usr/share/virt-manager/virtManager/engine.py", line 803, in _show_vm_helper
 details = self._get_details_dialog(uri, uuid)
 File "/usr/share/virt-manager/virtManager/engine.py", line 760, in _get_details_dialog
 obj = vmmDetails(con.get_vm(uuid))
 File "/usr/share/virt-manager/virtManager/details.py", line 530, in __init__
 self.init_details()
 File "/usr/share/virt-manager/virtManager/details.py", line 990, in init_details
 for name in [c.model for c in cpu_values.cpus]:
AttributeError: 'NoneType' object has no attribute 'cpus'
[Tue, 04 Mar 2014 22:13:31 virt-manager 21019] DEBUG (error:84) error dialog message:
summary=Error launching details: 'NoneType' object has no attribute 'cpus'
details=Error launching details: 'NoneType' object has no attribute 'cpus'

I opened a bug report and the fix was committed upstream today. If you want to make these updates to your Fedora 20 server before the update package is available, just snag the three RPM’s from koji and install them:

mkdir /tmp/virt-manager
cd /tmp/virt-manager
wget http://kojipkgs.fedoraproject.org/packages/virt-manager/1.0.0/4.fc20/noarch/virt-install-1.0.0-4.fc20.noarch.rpm
wget http://kojipkgs.fedoraproject.org/packages/virt-manager/1.0.0/4.fc20/noarch/virt-manager-1.0.0-4.fc20.noarch.rpm
wget http://kojipkgs.fedoraproject.org/packages/virt-manager/1.0.0/4.fc20/noarch/virt-manager-common-1.0.0-4.fc20.noarch.rpm
yum localinstall *.rpm

UPDATE: Thanks to Cole’s comment below, you can actually pull in the RPM’s using koji directly:

koji download-build virt-manager-1.0.0-4.fc20

Installing Xen on Fedora 20

major@mhtx.net (Major Hayden) — Fri, 28 Feb 2014 03:43:27 +0000

I’ve written about installing Xen on Fedora 19 and earlier versions on this blog before. Let’s tackle it on Fedora 20.

Start with the Xen hypervisor and the basic toolset first:

yum -y install xen xen-hypervisor xen-libs xen-runtime
systemctl enable xend.service
systemctl enable xendomains.service

Get GRUB2 in order:

# grep ^menuentry /boot/grub2/grub.cfg | cut -d "'" -f2
Fedora, with Linux 3.13.4-200.fc20.x86_64
Fedora, with Linux 0-rescue-c9dcecb251df472fbc8b4e620a749f6d
Fedora, with Xen hypervisor
# grub2-set-default 'Fedora, with Xen hypervisor'
# grub2-editenv list
saved_entry=Fedora, with Xen hypervisor
# grub2-mkconfig -o /boot/grub2/grub.cfg

Now reboot. When the server restarts, verify that Xen is running:

# xm dmesg | head
 __ __ _ _ _____ _ ___ __ ____ ___
 \ \/ /___ _ __ | || | |___ / / | / _ \ / _| ___|___ \ / _ \
 \ // _ \ '_ \ | || |_ |_ \ | |_| (_) || |_ / __| __) | | | |
 / \ __/ | | | |__ _| ___) || |__\__, || _| (__ / __/| |_| |
 /_/\_\___|_| |_| |_|(_)____(_)_| /_(_)_| \___|_____|\___/

(XEN) Xen version 4.3.1 (mockbuild@[unknown]) (gcc (GCC) 4.8.2 20131212 (Red Hat 4.8.2-7)) debug=n Thu Feb 6 16:52:58 UTC 2014
(XEN) Latest ChangeSet:
(XEN) Bootloader: GRUB 2.00
(XEN) Command line: placeholder

As I’ve mentioned before, I enjoy using virt-manager to manage my VM’s. Let’s get started:

yum -y install virt-manager dejavu* xorg-x11-xauth
yum -y install libvirt-daemon-driver-network libvirt-daemon-driver-storage libvirt-daemon-xen
systemctl enable libvirtd.service
systemctl start libvirtd.service

By this point, you have the Xen hypervisor running and you have VM management tools available from virt-manager and libvirt. Enjoy!

PXE boot Fedora 19 using a Mikrotik firewall

major@mhtx.net (Major Hayden) — Tue, 23 Jul 2013 21:47:33 +0000

Outside of the RHCA exams, I haven’t configured a PXE system for my personal needs. A colleague demoed his PXE setup for me and I was hooked. Once I realized how much time I could save when I’m building and tearing down virtual machines, it made complete sense. This post will show you how to configure PXE and tftpd in Mikrotik’s RouterOS to boot and install Fedora 19 (as well as provide rescue environments).

The first thing you’ll need are a few files from a working Fedora installation. Install the syslinux-tftpboot package and grab the following files:

/tftpboot/pxelinux.0
/tftpboot/vesamenu.c32

You’ll also need a vmlinuz and initrd.img file from your favorite Fedora mirror (use the linked text here for F19 x86_64 or look in the os/images/pxeboot directory on the mirror for your architecture).

When you have your four files, create a directory on the Mikrotik via FTP called tftp, and upload those to your Mikrotik. Your directory should look something like this:

 ls tftp/
-rw-rw---- 1 root root 155792 Jul 23 00:01 vesamenu.c32
-rw-rw---- 1 root root 5055896 Jul 22 23:41 vmlinuz
-rw-rw---- 1 root root 32829968 Jul 22 23:42 initrd.img
-rw-rw---- 1 root root 26460 Jul 22 23:37 pxelinux.0

Within the tftp directory, make a directory called pxelinux.cfg. Add a file called default inside the pxelinux.cfg directory with these contents:

default vesamenu.c32
prompt 0
timeout 600

display boot.msg

label linux
 menu label ^Install or upgrade an existing system
 kernel vmlinuz
 append initrd=initrd.img repo=http://mirrors.kernel.org/fedora/releases/19/Fedora/x86_64/os/ ks=http://example.com/kickstart.ks ip=eth0:dhcp
label vesa
 menu label Install system with ^basic video driver
 kernel vmlinuz
 append initrd=initrd.img xdriver=vesa nomodeset
label rescue
 menu label ^Rescue installed system
 menu default
 kernel vmlinuz
 append initrd=initrd.img repo=http://mirrors.kernel.org/fedora/releases/19/Fedora/x86_64/os/ rescue ip=eth0:dhcp
label local
 menu label Boot from ^local drive
 localboot 0xffff

Be sure to adjust the ip= and repo= arguments to fit your server. Keep in mind that from Fedora 17 on, you’ll need to use the dracut syntax for anaconda boot options. Once that’s done, you’re ready to configure the Mikrotik firewall, so get logged into the firewall over ssh.

We need to set some network options for our Mikrotik’s DHCP server:

/ip dhcp-server network
set 0 boot-file-name=pxelinux.0 next-server=192.168.25.1

The value for next-server= should be the gateway address for your internal network (the Mikrotik’s internal IP).

Next, we need to configure the tftp server so that it serves up files to our internal network:

/ip tftp
add ip-addresses=192.168.25.0/24 real-filename=tftp/pxelinux.0 req-filename=pxelinux.0
add ip-addresses=192.168.25.0/24 real-filename=tftp/pxelinux.cfg/default req-filename=pxelinux.cfg/default
add ip-addresses=192.168.25.0/24 real-filename=tftp/vmlinuz req-filename=vmlinuz
add ip-addresses=192.168.25.0/24 real-filename=tftp/vesamenu.c32 req-filename=vesamenu.c32
add ip-addresses=192.168.25.0/24 real-filename=tftp/initrd.img req-filename=initrd.img

Now it’s time to test it! If you’re using a physical machine, double check your BIOS to verify that PXE boot is enabled for your ethernet interface. Most modern chipsets have support for it, but be sure to check that it’s enabled. You may have to reboot after enabling it in the BIOS for the ethernet BIOS to be included.

If you’re using a virtual machine, just start up virt-manager and choose Network Boot (PXE) from the installation options:

Once the VM boots, you’ll be sent straight to the PXE boot screen:

TAKE NOTE! In the pxelinux.cfg/default file, I set rescue mode to boot as the default option. This will prevent a situation where you forget to remove PXE from a system’s boot order and accidentally re-kickstart over the live system.

The installer should now boot up normally and you can install your Fedora system via kickstart or via the anaconda interface.

Boot VM’s with virt-manager and libvirt with ISO’s stored remotely via samba/cifs

major@mhtx.net (Major Hayden) — Sun, 07 Jul 2013 01:51:10 +0000

Pairing virt-manager with KVM makes booting new VM’s pretty darned easy. I have a QNAP NAS at home with a bunch of ISO’s stored in share available to guests and I wanted to use that with libvirt to boot new VM’s. (By the way, if you’re looking for an off-the-shelf NAS that is built with solid hardware and pretty reliable software, try one of the QNAP devices. You still get access to many of the usual commands that you would normally find on a Linux box for emergencies. More on that in a later post.)

The first step was creating a mountpoint and configuring the mount in /etc/fstab:

# mkdir /mnt/iso
# grep qemu /etc/passwd
qemu❌107:107:qemu user:/:/sbin/nologin
# echo "//qnap/ISO /mnt/iso cifs _netdev,guest,uid=107,gid=107,defaults 0 0" >> /etc/fstab
# mount /mnt/iso

My QNAP is already in /etc/hosts so I didn’t need to specify the IP in the file. Adding _netdev ensures that the network will be up before the mount is made. The guest option ensures that I won’t be prompted for credentials and the uid=107,gid=107 mounts the share as the qemu user. If you forget this, virt-manager will throw some ugly permissions errors from libvirt.

From there, I had another permissions error and I suspected that SELinux was preventing libvirt from accessing the files in the share. A quick check of /var/log/messages revealed that I was right:

Jul 6 16:12:51 nuc1 setroubleshoot: SELinux is preventing /usr/bin/qemu-system-x86_64 from open access on the file /mnt/iso/livecd.iso. For complete SELinux messages. run sealert -l c1c80b2c-b5df-4114-86c7-ffee98274552

Here’s the output from sealert:

# sealert -l c1c80b2c-b5df-4114-86c7-ffee98274552
SELinux is preventing /usr/bin/qemu-system-x86_64 from open access on the file /mnt/iso/livecd.iso.

***** Plugin catchall_boolean (89.3 confidence) suggests *******************

If you want to allow virt to use samba
Then you must tell SELinux about this by enabling the 'virt_use_samba' boolean.
You can read 'None' man page for more details.
Do
setsebool -P virt_use_samba 1

The fix is a quick one:

# setsebool -P virt_use_samba 1

You should be all set after that. Press “Browse Local” in virt-manager when you look for your ISO to boot the virtual machine and navigate over to /mnt/iso for your list of ISO’s.

Installing the Xen hypervisor on Fedora 19

major@mhtx.net (Major Hayden) — Mon, 03 Jun 2013 04:27:43 +0000

It’s been a little while since I last posted about installing Xen on Fedora, so I figured that Fedora 19’s beta release was as good a time as any to write a new post. To get started, you’ll need to get Fedora 19 installed on your favorite hardware (or virtual machine).

Install the Xen hypervisor and tools. Also, ensure that both of the necessary daemons are running on each boot:

yum -y install xen xen-hypervisor xen-libs xen-runtime
chkconfig xend on
chkconfig xendomains on

You’ll notice that I didn’t start the daemons quite yet. We will need the xen hypervisor running before they will be of any use.

Now, let’s configure GRUB2. I wrote a quick post about these steps last year. The Xen kernel entry should already be configured (by grubby), but it’s not the default. Fixing that is a quick process:

# grep ^menuentry /boot/grub2/grub.cfg | cut -d "'" -f2
Fedora, with Linux 3.9.4-300.fc19.x86_64
Fedora, with Linux 0-rescue-4ea51ecfff4f4e64a5ec903c495ee5b6
Fedora, with Xen hypervisor
# grub2-set-default 'Fedora, with Xen hypervisor'
# grub2-editenv list
saved_entry=Fedora, with Xen hypervisor

At this point, you’re ready to reboot. After the reboot, verify that Xen is running:

# xm dmesg | head
 __ __ _ _ ____ ____ ____ __ _ ___
 \ \/ /___ _ __ | || | |___ \ |___ \ | ___| / _| ___/ |/ _ \
 \ // _ \ '_ \ | || |_ __) | __) |_|___ \ | |_ / __| | (_) |
 / \ __/ | | | |__ _| / __/ _ / __/|__|__) || _| (__| |\__, |
 /_/\_\___|_| |_| |_|(_)_____(_)_____| |____(_)_| \___|_| /_/

(XEN) Xen version 4.2.2 (mockbuild@phx2.fedoraproject.org) (gcc (GCC) 4.8.0 20130412 (Red Hat 4.8.0-2)) Fri May 17 19:39:53 UTC 2013
(XEN) Latest ChangeSet: unavailable
(XEN) Bootloader: GRUB 2.00
(XEN) Command line: placeholder

If you’re adventurous on the command line, you’re done here. However, I enjoy using virt-manager for quick access to virtual machines and I also like all of the scripting and remote administration capabilities that libvirt delivers. Let’s get the tools and daemons installed and running:

yum -y install virt-manager dejavu* xorg-x11-xauth
yum -y install libvirt-daemon-driver-network libvirt-daemon-driver-storage libvirt-daemon-xen
chkconfig libvirtd on
service libvirtd start

You’re now ready to use virt-manager to manage your virtual machines. Simply ssh to your hypervisor with X forwarding enabled (ssh -X hypervisor.mydomain.com) and run virt-manager. You won’t have a virtual network or bridge to use for virtual machines quite yet. You have two options: NAT your VM’s or configure a network bridge. I prefer the bridge but you may require something different in your environment.

For the NAT option (the easiest for beginners):

yum -y install libvirt-daemon-config-network libvirt-daemon-config-nwfilter
service libvirtd restart

For the network-bridge option, you’ll need to adjust your network scripts to create a bridge and add your primary network interface to the bridge. That’s a bit outside the scope of this post, but the Fedora Wiki and HowtoForge (ignore the KVM parts of their guide).

You now have a working Xen installation on Fedora 19!

FOR THOSE WHO EMBRACE SECURITY:

If you run SELinux in Enforcing mode, there’s still a lingering issue where SELinux prevents python (running under xend) from talking to block devices (like logical volumes). I opened a bug about a similar problem before but I need to open another one for the block device issue. If you’re itching for a workaround, you can force SELinux into permissive mode for the xend_t context only:

yum -y install selinux-policy-devel
semanage permissive -a xend_t

That’s not the best option for now, but it’s certainly better than setenforce 0. ;)

Migrate KVM virtual machines from CentOS 6 to Fedora 18 without the luxury of shared storage

major@mhtx.net (Major Hayden) — Wed, 22 May 2013 15:15:36 +0000

I’ve converted one of my KVM hypervisors from CentOS 6 to Fedora 18 and now comes the task of migrating my virtual machines off of my single remaining CentOS 6 hypervisor. This is definitely on a budget, so there’s no shared storage to make this process easier.

Here’s how I did it:

Migrate the logical volume

My first VM to migrate is my Fedora development VM where I build and test new packages. I have a 10G logical volume on the old node:

[root@helium ~]# lvs /dev/mapper/vg_helium-fedora--dev
 LV VG Attr LSize Pool Origin Data% Move Log Copy% Convert
 fedora-dev vg_helium -wi-a--- 10.00g

I made a 10G logical volume on the new hypervisor:

[root@hydrogen ~]# lvcreate -n fedora-dev -L10G vg_hydrogen
 Logical volume "fedora-dev" created

After getting ssh keys set up between both hypervisors and installing pv (to track progress), I started the storage migration over ssh:

dd if=/dev/mapper/vg_helium-fedora--dev | pv | ssh hydrogen dd of=/dev/mapper/vg_hydrogen-fedora--dev

Luckily it was only a 10GB logical volume so it transferred over in a few minutes.

Dump and adjust the source VM’s XML

On the source server, I dumped the VM configuration to an XML file and copied it to the new host:

virsh dumpxml fedora-dev > fedora-dev.xml
scp fedora-dev.xml hydrogen:

Before importing the XML file on the new host, there are some adjustments that need to be made. First off was an adjustment of the storage volume since the new host had the same logical volume name but a different volume group (the source line):

 <disk type='block' device='disk'>
 <driver name='qemu' type='raw' cache='none' io='native'></driver>
 <source dev='/dev/vg_hydrogen/fedora-dev'/>
 <target dev='vda' bus='virtio'></target>


<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'>

</address>
</disk>

Also, there’s a mismatch with the machine type (not architecture) between CentOS 6 and Fedora 18. I dumped the XML from a VM running on the Fedora 18 hypervisor and compared the machine type to my old CentOS VM’s XML (the XML from the CentOS VM is on top):

- <type arch='x86_64' machine='rhel6.3.0'>hvm</type>
+ <type arch='x86_64' machine='pc-1.2'>hvm</type>

I replaced rhel6.3.0 with pc-1.2. If you forget this step, your VM won’t start. You’ll get some errors about a mismatched machine type before the VM boots.

There’s one last fix: the path to the qemu-kvm emulator:

- <emulator>/usr/libexec/qemu-kvm</emulator>
+ <emulator>/usr/bin/qemu-kvm</emulator>

Replace /usr/libexec/qemu-kvm with /usr/bin/qemu-kvm and save your XML file.

Import the VM configuration and launch the VM

Importing the VM on the Fedora 18 hypervisor was easy:

virsh define fedora-dev.xml

That causes the configuration to load into libvirt and it should appear in virt-manager or virsh list by this point. If not, double check your previous steps and look for error messages in your logs. That doesn’t actually start the virtual machine, so I started it on the command line:

virsh start fedora-dev

Within a few moments, the VM was up and responding to pings.

It’s a good idea to hop into virt-manager and verify that the VM configuration is what you expect. Some configuration options don’t line up terribly well between CentOS 6 and Fedora 18. You might need to adjust a few to match the performance you expect to see.

Changing your ssh server’s port from the default: Is it worth it?

major@mhtx.net (Major Hayden) — Wed, 15 May 2013 04:43:41 +0000

Changing my ssh port from the default port (22) has been one of my standard processes for quite some time when I build new servers or virtual machines. However, I see arguments crop up regularly about it (like this reddit thread or this other one).

Before I go any further, let’s settle the “security through obscurity” argument. (This could probably turn into its own post but I’ll be brief for now.) Security should always be applied in layers. This provides multiple levels of protection from initial attacks, like information gathering attempts or casual threats against known vulnerabilities. In addition, these layers of security should be applied within the environment so that breaking into one server after getting a pivot point in the environment should be just as difficult (if not more difficult) than the original attack that created the pivot point. If “security through obscurity” tactics make up one layer of a multi-layered solution, I’d encourage you to obscure your environment as long as it doesn’t affect your availability.

The key takeaway is:

Security through obscurity is effective if it’s one layer in a multi-layer security solution

Let’s get back to the original purpose of the post.

The biggest benefit to changing the port is to avoid being seen by casual scans. The vast majority of people hunting for any open ssh servers will look for port 22. Some will try the usual variants, like 222 and 2222, but those are few and far between. I ran an experiment with a virtual machine exposed to the internet which had sshd listening on port 22. The server stayed online for one week and then I changed the ssh port to 222. The number of attacks dropped by 98%. Even though this is solely empirical evidence, it’s clear that moving off the standard ssh port reduces your server’s profile.

If it’s more difficult to scan for your ssh server, your chances of being attacked with an ssh server exploit are reduced. A determined attacker can still find the port if they know your server’s IP address via another means (perhaps via a website you host) and they can launch attacks once they find it. Paranoid server administrators might want to check into port knocking to reduce that probability even further.

Remembering the non-standard ssh port can be annoying, but if you have a standard set of workstations that you use for access your servers, just utilize your ~/.ssh/config file to specify certain ports for certain servers. For example:

Host *.mycompany.com
 Port 4321

Host nonstandard.mypersonalstuff.com
 Port 2345

Host *.mypersonalstuff.com
 Port 5432

If you run into SELinux problems with a non-standard ssh port, there are plenty of guides on this topic.. The setroubleshoot-server package helps out with this as well.

# semanage port -a -t ssh_port_t -p tcp 4321
# semanage port -l | grep ssh
ssh_port_t tcp 4321,22

Here is my list of ssh lockdown practices when I build a new server:

Update the ssh server package and ensure that automatic updates are configured
Enable SELinux and allow a non-standard ssh port
Add my ssh public key to the server
Disable password logins for ssh
Adjust my AllowUsers setting in sshd_config to only allow my user
Disable root logins
For servers with sensitive data, I install fail2ban

virt-manager won’t release the mouse when using ssh forwarding from OS X

major@mhtx.net (Major Hayden) — Wed, 20 Mar 2013 05:26:56 +0000

The latest versions of virt-manager don’t release the mouse pointer when you’re doing X forwarding to a machine running OS X. This can lead to a rather frustrating user experience since your mouse pointer is totally stuck in the window. Although this didn’t affect me with CentOS 6 hosts, Fedora 18 hosts were a problem.

There’s a relatively elegant fix from btm.geek that solved it for me. On your Mac, exit X11/Xquartz and create an ~/.Xmodmap file containing this:

clear Mod1
keycode 66 = Alt_L
keycode 69 = Alt_R
add Mod1 = Alt_L
add Mod1 = Alt_R

Start X11/Xquartz once more and virt-manager should release your mouse pointer if you hold the left control key and left option at the same time.

Late night virtualization frustration with kvm

major@mhtx.net (Major Hayden) — Wed, 20 Mar 2013 05:07:21 +0000

I dragged out an old Aopen MP57-D tonight that was just sitting in the closet and decided to load up kvm on Fedora 18. I soon found myself staring at a very brief error message upon bootup:

kvm: disabled by bios

After a reboot, the BIOS screen was up and I saw that Virtualization and VT-d were both enabled. Trusted execution (TXT) was disabled, so I enabled it for kicks and rebooted. Now I had two errors:

kvm: disable TXT in the BIOS or activate TXT before enabling KVM
kvm: disabled by bios

Time for another trip to the BIOS. I disabled TXT, rebooted, and I was back to the same error where I first started. A quick check of /proc/cpuinfo showed that I had the right processor extensions. Even the output of lshw showed that I should be ready to go. Some digging in Google led me to a blog post for a fix on Dell Optiplex hardware.

The fix was to do this:

Within the BIOS, disable virtualization, VT-d, and TXT
Save the BIOS configuration, reboot, and pull power to the computer at grub
Within the BIOS, enable virtualization and VT-d but leave TXT disabled
Save the BIOS configuration, reboot, and pull power to the computer at grub
Boot up the computer normally

Although it seems a bit archaic, this actually fixed my problem and set me on my way.

SELinux, Xen, and block devices in Fedora 17

major@mhtx.net (Major Hayden) — Tue, 10 Jul 2012 05:05:33 +0000

If you try to run Xen without libvirt on Fedora 17 with SELinux in enforcing mode, you’ll be butting heads with SELinux in no time. You’ll probably be staring at something like this:

# xm create -c fedora17
Using config file "/etc/xen/fedora17".
Error: Disk isn't accessible

If you have setroubleshoot and setroubleshoot-server installed, you should have a friendly message in /var/log/messages telling you the source of the problem:

setroubleshoot: SELinux is preventing /usr/bin/python2.7 from read access on the blk_file dm-1.
For complete SELinux messages. run sealert -l 4d890105-d9a4-4b3e-a674-ba7e952942dc

The Xen daemon (the python process mentioned in the SELinux denial) is running with a context type of xend_t but the block device I’m trying to use for the VM has fixed_disk_device_t:

# ps axZ | grep xend
system_u:system_r:xend_t:s0 953 ? SLl 0:40 /usr/bin/python /usr/sbin/xend
# ls -alZ /dev/dm-1
brw-rw----. root disk system_u:object_r:fixed_disk_device_t:s0 /dev/dm-1

SELinux isn’t going to allow this to work. However, even if we fix this, SELinux will balk about three additional issues and we’ll need to adjust the contexts on every new fixed block device we make. To get over the hump, change the context type on your block device to xen_image_t and re-run the xm create:

# chcon -t xen_image_t /dev/dm-1
# ls -alZ /dev/dm-1
brw-rw----. root disk system_u:object_r:xen_image_t:s0 /dev/dm-1
# xm create -c fedora17
Using config file "/etc/xen/fedora17".
Error: out of pty devices

You’ll find three new denials in /var/log/messages:

setroubleshoot: SELinux is preventing /usr/bin/python2.7 from read access on the file group.
For complete SELinux messages. run sealert -l b1392df4-dda4-4b82-914c-1e20c62fc898
setroubleshoot: SELinux is preventing /usr/bin/python2.7 from setattr access on the chr_file 1.
For complete SELinux messages. run sealert -l 3e09edc3-aeb7-49f5-96e1-d8148afda48f
setroubleshoot: SELinux is preventing /usr/bin/python2.7 from execute access on the file pt_chown.
For complete SELinux messages. run sealert -l 86395f09-5f33-4f66-8d02-519b61e54139

As much as it pains me to suggest it, you can create a custom module to allow all four of these operations by xend:

# grep xend /var/log/audit/audit.log | audit2allow -M custom_xen
WARNING: Policy would be downgraded from version 27 to 26.
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i custom_xen.pp

# semodule -i custom_xen.pp

You should now be able to start your VM without any complaints from SELinux. I’ll reiterate that this isn’t ideal, but it’s the best balance of security and convenience that I’ve found so far.

Installing XenServer 6.0.2 on an AOpen MP57

major@mhtx.net (Major Hayden) — Mon, 12 Mar 2012 17:00:56 +0000

Getting XenServer installed on some unusual platforms takes a bit of work and the AOpen MP57 is a challenging platform for a XenServer 6.0.2 installation.

My MP57 box came with the i57QMx-vP motherboard. If yours came with something else, this post may or may not work for you.

You’ll need the XenServer 6 installation ISO burned to a CD to get started. Boot the CD in your MP57 and wait for the initial boot screen to appear. Type safe at the prompt and press enter. Go through the normal installation steps and reboot.

After the reboot, you’ll notice that there’s no video output for dom0. Hop on another nearby computer and ssh to your XenServer installation using the root user and the password that you set during the installation process. Open up /boot/extlinux.conf in your favorite text editor and make sure the label xe section looks like this:

label xe
 # XenServer
 kernel mboot.c32
 append /boot/xen.gz mem=1024G dom0_max_vcpus=4 dom0_mem=752M lowmem_emergency_pool=1M crashkernel=64M@32M acpi=off console=vga --- /boot/vmlinuz-2.6-xen root=LABEL=root-aouozuoo ro xencons=hvc console=hvc0 console=tty0 vga=785 --- /boot/initrd-2.6-xen.img

The console=vga adjustment ensures that the dom0 console is piped to the vga output and acpi=off fixes the lockup that will occur when the vga output is sent to your display. I also removed splash and quiet from the kernel line so that I could see all of the boot messages in detail.

Preparing for Red Hat Exams

major@mhtx.net (Major Hayden) — Tue, 28 Feb 2012 21:35:28 +0000

I originally wrote this post for the Rackspace Blog but I’ve posted it here just in case anyone following my blog’s feed finds it useful. Feel free to share your feedback!

Getting yourself ready for any type of examination is usually a stressful experience that involves procrastination and some late nights leading up to the test. Every time I take one, I always say to myself, “I’m really going to get ahead of this next time and study early. This last minute stuff is terrible.” But I always forget all of this as the next exam rolls around.

Quick note: As you read through the remainder of the post, you may wonder why some of it is a bit vague. Every Red Hat test taker is under a NDA to prevent disclosure of test information that may reduce the security of the exam itself. Penalties start with losing credit for the exams previously taken and they can escalate up to legal action. I hope you’ll understand why I’m not able to go into details about certain portions of the Red Hat examinations.

I’ve taken seven Red Hat exams already: two for the RHCE and five for the RHCA. These tests certainly aren’t easy, but there are some good guidelines and tips you can use to make your studying efforts less stressful and more productive. Without further ado, here are my recommendations for prospective Red Hat examinees:

Build a flexible study environment

This is critical. You’ll need some spare servers or some available virtual machines to practice the objectives on each exam. However, don’t feel like you need to spend the money on a Red Hat subscription to get your studying done. Most of the test objectives on the majority of exams can be completed with very similar Linux distributions, like Scientific Linux or CentOS. Look for a version of the distribution that is closest to what you’ll be tested on at exam time. Your study environment should meet some basic criteria:

You should be able to quickly build and tear down servers or virtual machines
Keep the latency to your environment low to avoid getting frustrated
Use applications like VirtualBox, VMWare Fusion/Workstation to practice on your own computer
Consider using VMs from cloud providers if you’re under a time crunch

Some exams may require some bare-metal access to the server itself (especially EX442), so keep that in mind when you’re looking for a good practice environment. You may need some specific network or storage setups for some exams (as with EX436). If you’re not sure what you need, be sure to ask your instructor or someone else you know who has taken the exam already.

Prioritize doing over reading

The Red Hat exams are all hands-on, practical exams. You won’t find any essays or multiple-choice questions in these exams. Although the materials from Red Hat are full of good information, reading this information can only get you so far. You need to practice setting up the services on your own to be fully prepared for the test. If you’re not pressed for time, reading through the book can give you some details about the lab sequences, which you might miss by solely reading through labs themselves.

Research the why, not the what, to remember

This is especially important for the RHCA exam track. You may find that there is a ton of material to cover for the exam and that it’s difficult to remember each command to bring a certain service online or to repair a problem. Instead of thinking through the problem as “first, I do this, then I do this”, try to understand why each step is important in the first place.

Here’s a good example. I’ll be the first one to admit that Kerberos drives me crazy. I’ve even written posts about it. The commands seemed really archaic, the daemons didn’t make sense, and the lack of readline support in the Kerberos tools made me want to throw my computer out the window (come on, MIT!). I put my class materials aside, went to Google in a browser, and started researching Kerberos.

I read some of MIT’s documentation, ventured over to Wikipedia, and poked at some of the documentation within the Kerberos RPM packages. After a while, I began to realize how it all fit together. “Okay,” I thought to myself, “I need principals in a keytab to do these things, but I need to have a database for the admin stuff first.” Suddenly, the order of things in my head wasn’t just memorized any longer. The process of operations seemed to make logical sense because I fully understood how the pieces of a Kerberos infrastructure fit together.

If you start to get discouraged, take a break and learn more about why you’re doing what you’re doing. Once it becomes second nature, working through the problems on the exam becomes much easier.

Lean on your available resources

Don’t forget that there are other knowledgeable folks available to talk to when you get bogged down. Lean on other RHCE’s, RHCA’s, or experienced Linux users to get the answers or explanations you need. If you already have a Red Hat certification, head over to the Red Hat Certification Forums and meet up with other examinees that are discussing test preparation.

Also, you’ll find some knowledgeable (but sometimes snarky or quirky) people on IRC who are eager to point you in the right direction. Try the #rhel, #centos, or #fedora channels if you’re struggling through the configuration of a certain service. Many Linux users may roll their eyes about it, but Twitter is also a pretty good way to reach out to people who have a lot of Linux experience.

Summary

Remember to lean on the knowledge of others, get hands-on with the test objectives and do your research when you’re frustrated. The exams from Red Hat are generally difficult and cover a lot of material, but with the right amount of preparation and determination you can pass the exams and get the certifications you want.

Xen 4.1 on Fedora 15 with Linux 3.0

major@mhtx.net (Major Hayden) — Sat, 06 Aug 2011 04:34:06 +0000

If you haven’t noticed already, full Xen dom0 support was added in the Linux 3.0 kernel. This means there’s no longer a need to drag patches forward from old kernels and work from special branches and git repositories when building a kernel for dom0.

Something else you might not have noticed is that the Fedora kernel team has quietly slipped Linux 3.0 into Fedora 15’s update channels in disguise. Click that link, scroll down, and you’ll see “Rebase to 3.0. Version reports as 2.6.40 for compatibility with older userspace.” Although I’m not a fan of calling something what it isn’t (2.6.40 doesn’t exist on kernel.org), I can understand some of the reasoning behind the choice.

This change makes the Xen installation on Fedora 15 pretty trivial. To get started, update your kernel to the latest if you’re not already on Fedora’s 2.6.40 kernels:

yum -y upgrade kernel

We need three more packages (quite a few dependencies will roll in with them):

yum -y install xen libvirt python-virtinst

The xen package reels in the hypervisor itself along with libraries and command line tools (like xl and xm). Libvirt gives us easy access to VM management with the virsh command and python-virtinst gives us the handy virt-install command to make OS installations easy.

Once those packages are installed, we need to make some adjustments in your grub configuration. Open /boot/grub/menu.lst in your text editor of choice and add something like this at the bottom:

title Fedora + Xen (2.6.40-4.fc15.x86_64)
 root (hd0,1)
	kernel /boot/xen.gz
 module /boot/vmlinuz-2.6.40-4.fc15.x86_64 ro root=/dev/sda1
 module /boot/initramfs-2.6.40-4.fc15.x86_64.img

Ensure that the root (hd0,1) is applicable to your system (adjust it if it isn’t). Also, check the kernel version to ensure it matches your installed kernel and adjust the root= portion to match your root volume. Flip the default line to a value which will boot your new grub entry and ensure the timeout is set to a reasonable number if you need to temporarily switch back to your original grub entry at boot time. (Hey, we all make mistakes.)

I take one extra precaution and change the UPDATEDEFAULT=yes line to no in /etc/sysconfig/kernel. This ensures that future kernel updates don’t trample the entry you’ve just made. Keep in mind that you’ll need to manually update your grub configuration when you do kernel upgrades later.

Cross your fingers and reboot. If your system doesn’t reboot properly, reboot it again and choose your old kernel from the grub menu. Double-check your configuration for fat-fingering and give it another try. If your system boots and pings but you have no output via a monitor, don’t fret. There’s a patch for the problem which should appear soon in Linux 3.0. The impatient can snag a kernel source RPM, add the patch file, and build a local kernel (or you can download my local build from when I did it).

[root@xenbox ~]# xm dmesg | head -n 5
 __ __ _ _ _ _ ____ __ _ ____
 \ \/ /___ _ __ | || | / | / | |___ \ / _| ___/ | ___|
 \ // _ \ '_ \ | || |_ | | | |__ __) | | |_ / __| |___ \
 / \ __/ | | | |__ _|| |_| |__/ __/ _| _| (__| |___) |
 /_/\_\___|_| |_| |_|(_)_(_)_| |_____(_)_| \___|_|____/

Once you’re done with that, make sure libvirtd is running:

/etc/init.d/libvirtd start; chkconfig libvirtd on

Try installing a VM:

virt-install \
 --paravirt \
 --name=testvm \
 --ram=512 \
 --vcpus=4 \
 --file /dev/vmstorage/testvm \
 --graphics vnc,port=5905 --noautoconsole \
 --autostart --noreboot \
 --location=http://mirrors.kernel.org/debian/dists/squeeze/main/installer-amd64/

You should have a VM installation underway pretty quickly and it will be visible via port 5905 on the local host. Enjoy the power and freedom of your brand new type 1 hypervisor.

Installing Xen 4 on Fedora 13

major@mhtx.net (Major Hayden) — Fri, 10 Sep 2010 13:56:49 +0000

Installing Xen can be a bit of a challenge for a beginner and it’s made especially difficult by distribution vendors who aren’t eager to include it in their current releases. I certainly don’t blame the distribution vendors for omitting it; the code to support Xen’s privileged domain isn’t currently in upstream kernels.

However, Pasi Kärkkäinen has written a detailed walkthrough about how to get Xen 4 running on Fedora 13. Although there are quite a few steps involved, it’s worked well for me so far.

Legacy tty1 and block device support for Xen guests with pvops kernels

major@mhtx.net (Major Hayden) — Fri, 14 May 2010 13:24:34 +0000

The discussions about the paravirt_ops, or “pvops”, support in upstream kernels at Xen Summit 2010 last month really piqued my interest.

Quite a few distribution maintainers have gone to great lengths to keep Xen domU support in their kernels and it’s been an uphill battle. Some kernels, such as Ubuntu’s linux-ec2 kernels, have patches from 2.6.18 dragged forward into 2.6.32 and even 2.6.33. It certainly can’t be enjoyable to keep dragging those patches forward into new kernel trees.

The paravirt_ops support for Xen guests was added in 2.6.23 and continues to be included and improved in the latest kernel trees. However, there are two significant problems with these new kernels if you’re trying to work with legacy environments:

the console is on hvc0, not tty1
block devices are now /dev/xvdX rather than /dev/sdX

If you only have a few guests, these changes are generally pretty easy. Switching the console just requires some changes to your inittab or upstart configurations. Changing the block device names requires changes to the guest’s Xen configuration file and /etc/fstab within the guest itself.

Considering the amount of environments I work with daily at Rackspace, changing the guest configuration is definitely not an option. I needed a way to keep the console and block devices unchanged so that our customers could have a consistent experience on our infrastructure.

Luckily, Soren Hansen offered to pitch in and a solution became apparent. Through some relatively small patches, the legacy console and block device support was available in the latest 2.6.32 version (2.6.32.12 as of this post’s writing).

So far, I’ve tested x86_64 and i386 versions of 2.6.32.12 with the console and block device patches. It’s gone through its paces on Xen 3.0.3, 3.1.2, 3.3.0 and 3.4.2. All revisions of Fedora, CentOS, Ubuntu, Debian, Gentoo and Arch made within the last two years are working well with the new kernels.