Centos on Major Hayden

Texas Linux Fest 2024 recap 🤠

major@mhtx.net (Major Hayden) — Tue, 16 Apr 2024 00:00:00 +0000

The 2024 Texas Linux Festival just ended last weekend and it was a fun event as always. It’s one my favorite events to attend because it’s really casual. You have plenty of opportunities to see old friends, meet new people, and learn a few things along the way.

I was fortunate enough to have two talks accepted for this year’s event. One was focused on containers while the other was a (very belated) addition to my impostor syndrome talk from 2015.

This was also my first time building slides with reveal-md, a “batteries included” package for making reveal.js slides. Nothing broke too badly and that was a relief.

Containers talk

I’ve wanted to share more of what I’ve done with CoreOS in low-budget container deployments and this seemed like a good time to share it with the world out loud. My talk, Automated container updates with GitHub and CoreOS, walked the audience through how to deploy containers on CoreOS, keep them updated, and update the container image source.

My goal was to keep it as low on budget as possible. Much of it was centered around a stack of caddy, librespeed, and docker-compose. All of it was kept up to date with watchtower.

My custom Caddy container needed support for Porkbun’s DNS API and I used GitHub Actions to build that container and serve it to the internet using GitHub’s package hosting. This also gave me the opportunity to share how awesome Porkbun is for registering domains, including their customized pig artwork for every TLD imaginable. 🐷

We had a great discussion afterwards about how CoreOS does indeed live on as Fedora CoreOS.

Tech career talk

This talk made me nervous because it had a lot of slides to cover, but I also wanted to leave plenty of time for questions. Five tips for a thriving technology career built upon my old impostor syndrome talk by sharing some of the things I’ve learned over the year that helped me succeed in my career.

I managed to end early with time for questions, and boy did the audience have questions! 📣 Some audience members helped me answer some questions, too!

We talked a lot about office politics, tribal knowledge, and toxic workplaces. The audience generally agreed that most businesses tried to rub copious amounts of Confluence on their tribal knowledge problem, but it never improved. 😜

The room was full with people standing in the back and I’m tremendously humbled by everyone who came. I received plenty of feedback afterwards and that’s the best gift I could ever get. 🎁

Other talks

Anita Zhang had an excellent keynote talk on the second day about her unusual path into the world of technology. Her slides were pictures of her dog that lined up with various parts of her story. That was a great idea.

Kyle Davis offered talks on valkey and bottlerocket. There was plenty about the redis and valkey story that I didn’t know and the context was useful. It looks like you can simply drop valkey into most redis environments without much disruption.

Thomas Cameron talked about running OKD on Fedora CoreOS in his home lab. There were quite a few steps, but he did a great job of connecting the dots between what needed to be done and why.

Around the exhibit hall

I helped staff the Fedora/CoreOS booth and we had plenty of questions. Most questions were around the M1 Macbook running Asahi Linux that was on the table. 😉

There were still quite a few misconceptions around the CentOS Stream changes, as well as how AlmaLinux and Rocky Linux fit into the picture. Our booth was right next to the AlmaLinux booth and I had the opportunity to meet Jonathan Wright. That was awesome!

I can’t wait for next year’s event.

Build a Tailscale exit node with firewalld

major@mhtx.net (Major Hayden) — Thu, 27 Oct 2022 00:00:00 +0000

Once upon a time, I spent hours and hours fumbling through openvpn configurations, certificates, and firewalls to get VPNs working between servers. One small configuration error led to lots of debugging. Adding new servers meant wallowing through this process all over again.

A friend told me about Tailscale and it makes private networking incredibly simple.

Tailscale makes it easy to add nodes to a private network called a tailnet where they can communicate. In short, it’s a dead simple mesh network (with advanced capabilities if you’re interested).

This post covers how to create an exit node for your Tailscale network using firewalld Fedora, CentOS Stream, and Red Hat Enterprise Linux (RHEL).

What’s an exit node?

Every node on a Tailscale network, or tailnet, can communicate with each other¹. However, it can use be useful to use one of those nodes on the network as an exit node.

Exit nodes allow traffic to leave the tailnet and go out to other networks or the public internet. This allows you to join an untrusted network, such as a coffee shop’s wifi network, and send your traffic out through one of your tailnet nodes. It works more like a traditional VPN.

Tailscale does this by changing the routes on your device to use the exit node on your tailnet for all traffic. Creating an exit node involves some configuration on the node itself and within Tailscale’s administrative interface.

Deploy Tailscale

In this example, I’ll use Fedora 36, but these instructions work for CentOS Stream and RHEL, too.

Start by installing Tailscale on your system:

Let’s get this going on Fedora:

$ sudo dnf config-manager --add-repo https://pkgs.tailscale.com/stable/fedora/tailscale.repo
Adding repo from: https://pkgs.tailscale.com/stable/fedora/tailscale.repo
$ sudo dnf install tailscale
$ sudo systemctl enable --now tailscaled
$ sudo tailscale up

To authenticate, visit:

	https://login.tailscale.com/a/xxxxxx

Click the link to authorize the node to join your tailnet and it should appear in your list of nodes!

Trusting the tailnet

In my case, I treat my tailnet interfaces as trusted interfaces on each node. (This might not fit your use case, so please read the docs on Network access controls (ACLs) if you need extra security layers.)

Start by adding the new tailscale0 interface as a trusted interface:

# firewall-cmd --add-interface=tailscale0 --zone=trusted
success
# firewall-cmd --list-all --zone=trusted
trusted (active)
 target: ACCEPT
 icmp-block-inversion: no
 interfaces: tailscale0
 sources: 
 services: 
 ports: 
 protocols: 
 forward: yes
 masquerade: no
 forward-ports: 
 source-ports: 
 icmp-blocks: 
 rich rules:

This permits all traffic in and out of the tailscale interface.

Creating an exit node

Let’s reconfigure tailscale to allow this node to serve as an exit node:

# tailscale up --advertise-exit-node
Warning: IP forwarding is disabled, subnet routing/exit nodes will not work.
See https://tailscale.com/kb/1104/enable-ip-forwarding/

Uh oh. Tailscale is telling us that it’s happy to reconfigure itself, but we’re going to run into IP forwarding issues. Let’s see what our primary firewall zone has:

# firewall-cmd --list-all
public (active)
 target: default
 icmp-block-inversion: no
 interfaces: eth0
 sources: 
 services: dhcpv6-client mdns ssh
 ports: 
 protocols: 
 forward: yes
 masquerade: no
 forward-ports: 
 source-ports: 
 icmp-blocks: 
 rich rules:

We can enable masquerading in firewalld and it will take care of everything for us, including NAT rules and sysctl settings for IP forwarding.

# firewall-cmd --add-masquerade --zone=public
success
# firewall-cmd --list-all | grep masq
 masquerade: yes

Let’s try the Tailscale reconfiguration once more:

# tailscale up --advertise-exit-node
Warning: IPv6 forwarding is disabled.
Subnet routes and exit nodes may not work correctly.
See https://tailscale.com/kb/1104/enable-ip-forwarding/

We fixed the IPv4 forwarding, but IPv6 is still not configured properly. This was done automatically in the past, but firewalld does not automatically enable IPv6 forwarding in recent versions. Rich rules come to the rescue:

# firewall-cmd --add-rich-rule='rule family=ipv6 masquerade'
success
# sysctl -a | grep net.ipv6.conf.all.forwarding
net.ipv6.conf.all.forwarding = 1

One more try:

# tailscale up --advertise-exit-node

Success! But wait, we still can’t send traffic through the exit node until we authorize it in the Tailscale admin interface:

Go back to your machines list at Tailscale and find your exit node.
Right underneath the name of the node, you should see Exit Node followed by a circle with an exclamation point.
Click the three dots on the far right of that row and click Edit Route Settings….
When the modal appears, click the slider to the left of Use as exit node.

Now you can test your exit node from your mobile device by choosing an exit node via the settings menu. Another Linux machine can use it as an exit node as well just by running another tailscale up command:

# Start using an exit node.
sudo tailscale up --exit-node my-exit-node-name

# Stop using an exit node.
sudo tailscale up --exit-node ''

Tailscale offers complex access control lists (ACLs) that allow you to limit connectivity based on tons of factors. ↩︎

Build a custom CentOS Stream 9 cloud image

major@mhtx.net (Major Hayden) — Fri, 06 May 2022 00:00:00 +0000

This is my third post about Image Builder, so I guess you could say that I enjoy using it¹. It’s a great way to define a custom cloud image, build it, and (optionally) ship it to a supported cloud provider.

This post covers how to build a customized CentOS Stream 9 image along with a custom repository for additional packages. In this case, that’s Extra Packages for Enterprise Linux (EPEL).

Why do I need a custom image anyway?

Building your own image empowers you to choose which packages you want, which services run at boot time, and where you deploy your image. Some cloud providers may not have an image from the Linux distribution you enjoy most, or they might have an image with the wrong package set.

Some cloud providers build images with too many packages or too few. Sometimes they add configuration that doesn’t exist in the OS itself. I’ve even found some that alter cloud-init and force you to log in directly as the root user. 😱

I enjoy building my own images so I know exactly what it contains and I know that the configuration came from the OS itself.

First steps

This post uses CentOS Stream 9 as an example. You will need a physical host, virtual machine, or cloud instance running CentOS Stream 9 first. We start by installing some packages:

$ sudo dnf install osbuild-composer weldr-client

What do these packages contain?

osbuild-composer ensures you have osbuild, the low-level image build component, along with configuration and an osbuild-composer² worker that builds the image.
weldr-client contains the composer-cli command line tool that makes it easy to interact with osbuild-composer

One nice thing about this stack is that it starts via systemd’s socket activation and it only runs when you query it. Let’s start the socket now and ensure it comes up on a reboot:

$ sudo systemctl enable --now osbuild-composer.socket

Verify that the API is responding:

$ composer-cli status show
API server status:
 Database version: 0
 Database supported: true
 Schema version: 0
 API version: 1
 Backend: osbuild-composer
 Build: NEVRA:osbuild-composer-46-1.el9.x86_64

Adding EPEL

CentOS Stream 9 has most of the packages I want, but I really love this program called htop that displays resource usage and allows you to introspect certain processes or namespaces easily. This package is only available in the EPEL repository, so we need to add that one to our list of enabled repositories for image builds.

osbuild-composer comes with its own set of repositories in the package and does not use the system’s repositories. You can list all the enabled repositories that it knows about:

# composer-cli sources list
AppStream
BaseOS
RT

If we want to add EPEL, we can dump the configuration from one of these to a file and edit it:

# composer-cli sources info AppStream | tee epel.ini
check_gpg = true
check_ssl = true
id = "AppStream"
name = "AppStream"
rhsm = false
system = true
type = "yum-baseurl"
url = "https://composes.stream.centos.org/production/latest-CentOS-Stream/compose/AppStream/x86_64/os/"

After editing the EPEL repository file, it should look like this:

check_gpg = true
check_ssl = true
id = "EPEL9"
name = "EPEL9"
rhsm = false
system = false
type = "yum-baseurl"
url = "https://mirrors.kernel.org/fedora-epel/9/Everything/x86_64/"

You can use any mirror you prefer, but the kernel mirrors are very fast for me from most locations. Now we need to add this repository:

# composer-cli sources add epel.ini
# composer-cli sources list
AppStream
BaseOS
EPEL9
RT

We now have EPEL9 in our list. 🎉

Define our image

All image definitions, or blueprints, are in TOML format. Here’s my simple one for this post:

# Save this file as image.toml
name = "centos9"
description = "Major's awesome CentOS 9 image"
version = "0.0.1"

[[packages]]
name = "tmux"

[[packages]]
name = "vim"

# This is the one that comes from EPEL.
[[packages]]
name = "htop"

Now we push our blueprint and solve the dependencies to ensure we added our EPEL repository properly:

# composer-cli blueprints push image.toml
# composer-cli blueprints depsolve centos9
 -- SNIP --
 2:vim-filesystem-8.2.2637-16.el9.noarch
 which-2.21-27.el9.x86_64
 xz-5.2.5-7.el9.x86_64
 xz-libs-5.2.5-7.el9.x86_64
 zlib-1.2.11-33.el9.x86_64
 htop-3.1.2-3.el9.x86_64

And there’s htop at the end of the list! 🎉

Make the image

The fun part has arrived! Let’s build an image:

# composer-cli compose start centos9 ami --size=4096
Compose ca57fd64-11ea-41d4-b924-9b8f5bdcaf5e added to the queue

This command does a few things:

Starts an image build with our centos9 blueprint (from the name section of my image.toml file)
Outputs an image type that works well on AWS (Amazon Machine Image, or AMI)
Limits the image size to 4GB (be sure this is not too large for your preferred instance size)

🤔 Note that you do not need to set the size explicitly here, but I do it as a good measure. When your instance boots, cloud-init runs growpart to expand the storage to fit the disk size in your cloud instance.

💣 However, growpart will not shrink the disk at boot time. If you choose a size that is larger than the disk space in your cloud instance, you will likely see an error at provisioning time.

Let’s check the status after a few minutes:

# composer-cli compose status
ca57fd64-11ea-41d4-b924-9b8f5bdcaf5e FINISHED Fri May 6 16:16:08 2022 centos9 0.0.1 ami 4096

If you want to get a copy of the image and import it yourself into your favorite cloud, you can do that now:

# composer-cli compose image ca57fd64-11ea-41d4-b924-9b8f5bdcaf5e
ca57fd64-11ea-41d4-b924-9b8f5bdcaf5e-image.raw
# ls -alh ca57fd64-11ea-41d4-b924-9b8f5bdcaf5e-image.raw
-rw-------. 1 root root 2.7G May 6 16:20 ca57fd64-11ea-41d4-b924-9b8f5bdcaf5e-image.raw

You can also let osbuild-composer do this for you! I have one post on this blog about automatically uploading to AWS and another post on the Red Hat blog about doing the same with Azure.

I once worked on the team that makes Image Builder happen, so I may be a little bit biased. Enjoy the post anyway. 😉 ↩︎
Most people at Red Hat just call it composer, but I’ll use the full name here to avoid confusion. ↩︎

Allow a port range with firewalld

major@mhtx.net (Major Hayden) — Fri, 04 Jan 2019 00:00:00 +0000

Managing iptables gets a lot easier with firewalld. You can manage rules for the IPv4 and IPv6 stacks using the same commands and it provides fine-grained controls for various “zones” of network sources and destinations.

Quick example

Here’s an example of allowing an arbitrary port (for netdata) through the firewall with iptables and firewalld on Fedora:

## iptables
iptables -A INPUT -j ACCEPT -p tcp --dport 19999
ip6tables -A INPUT -j ACCEPT -p tcp --dport 19999
service iptables save
service ip6tables save

## firewalld
firewall-cmd --add-port=19999/tcp --permanent

In this example, firewall-cmd allows us to allow a TCP port through the firewall with a much simpler interface and the change is made permanent with the --permanent argument.

You can always test a change with firewalld without making it permanent:

firewall-cmd --add-port=19999/tcp
## Do your testing to make sure everything works.
firewall-cmd --runtime-to-permanent

The --runtime-to-permanent argument tells firewalld to write the currently active firewall configuration to disk.

Adding a port range

I use mosh with most of my servers since it allows me to reconnect to an existing session from anywhere in the world and it makes higher latency connections less painful. Mosh requires a range of UDP ports (60000 to 61000) to be opened.

We can do that easily in firewalld:

firewall-cmd --add-port=60000-61000/udp --permanent

We can also see the rule it added to the firewall:

# iptables-save | grep 61000
-A IN_public_allow -p udp -m udp --dport 60000:61000 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT
# ip6tables-save | grep 61000
-A IN_public_allow -p udp -m udp --dport 60000:61000 -m conntrack --ctstate NEW,UNTRACKED -j ACCEPT

If you haven’t used firewalld yet, give it a try! There’s a lot more documentation on common use cases in the Fedora firewalld documentation.

Install testing kernels in Fedora

major@mhtx.net (Major Hayden) — Wed, 28 Feb 2018 13:53:48 +0000

If you’re on the latest Fedora release, you’re already running lots of modern packages. However, there are those times when you may want to help with testing efforts or try out a new feature in a newer package.

Most of my systems have the updates-testing repository enabled in one way or another. This repository contains packages that package maintainers have submitted to become the next stable package in Fedora. For example, if there is a bug fix for nginx, the package maintainer submits the changes and publish a release. That release goes into the testing repositories and must sit for a waiting period or receive sufficient karma (“works for me” responses) to move into stable repositories.

Getting started

One of the easiest ways to get started is to allow a small amount of packages to be installed from the testing repository on a regular basis. Fully enabling the testing repository for all packages can lead to trouble on occasion, especially if a package maintainer discovers a problem and submits a new testing package.

To get started, open /etc/yum.repos.d/fedora-updates-testing.repo in your favorite text editor (using sudo). This file tells yum and dnf where it should look for packages. The stock testing repository configuration looks like this:

[updates-testing]
name=Fedora $releasever - $basearch - Test Updates
failovermethod=priority
#baseurl=http://download.fedoraproject.org/pub/fedora/linux/updates/testing/$releasever/$basearch/
metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-testing-f$releasever&arch=$basearch
enabled=0
repo_gpgcheck=0
type=rpm
gpgcheck=1
metadata_expire=6h
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
skip_if_unavailable=False

By default, the repository is not enabled (enabled=0).

In this example, let’s consider a situation where you want to test the latest kernel packages as soon as they reach the testing repository. We need to make two edits to the repository configuration:

enabled=1 - Allow yum/dnf to use the repository
includepkgs=kernel* - Only allow packages matching kernel* to be installed from the testing repository

The repository configuration should now look like this:

[updates-testing]
name=Fedora $releasever - $basearch - Test Updates
failovermethod=priority
#baseurl=http://download.fedoraproject.org/pub/fedora/linux/updates/testing/$releasever/$basearch/
metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-testing-f$releasever&arch=$basearch
enabled=1
repo_gpgcheck=0
type=rpm
gpgcheck=1
metadata_expire=6h
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
skip_if_unavailable=False
includepkgs=kernel*

Getting testing packages

Running dnf upgrade kernel* should now pull a kernel from the updates-testing repository. You can verify this by checking the Repository column in the dnf output.

If you feel more adventurous later, you can add additional packages (separated by spaces) to the includepkgs line. The truly adventurous users can leave the repo enabled but remove includepkgs altogether. This will pull all available packages from the testing repository as soon as they are available.

Package maintainers need feedback!

One final note: package maintainers need your feedback on packages. Positive or negative feedback is very helpful. You can search for the package on Bodhi and submit feedback there, or use the fedora-easy-karma script via the fedora-easy-karma package. The script will look through your installed package list and query you for feedback on each one.

Submitting lots of feedback can earn you some awesome Fedora Badges!

Photo credit: US Air Force

Ensuring keepalived starts after the network is ready

major@mhtx.net (Major Hayden) — Fri, 15 Dec 2017 21:18:37 +0000

After a recent OpenStack-Ansible (OSA) deployment on CentOS, I found that keepalived was not starting properly at boot time:

Keepalived_vrrp[801]: Cant find interface br-mgmt for vrrp_instance internal !!!
Keepalived_vrrp[801]: Truncating auth_pass to 8 characters
Keepalived_vrrp[801]: VRRP is trying to assign ip address 172.29.236.11/32 to unknown br-mgmt interface !!! go out and fix your conf !!!
Keepalived_vrrp[801]: Cant find interface br-mgmt for vrrp_instance external !!!
Keepalived_vrrp[801]: Truncating auth_pass to 8 characters
Keepalived_vrrp[801]: VRRP is trying to assign ip address 192.168.250.11/32 to unknown br-mgmt interface !!! go out and fix your conf !!!
Keepalived_vrrp[801]: VRRP_Instance(internal) Unknown interface !
systemd[1]: Started LVS and VRRP High Availability Monitor.
Keepalived_vrrp[801]: Stopped
Keepalived[799]: Keepalived_vrrp exited with permanent error CONFIG. Terminating

OSA deployments have a management bridge for traffic between containers. These containers run the OpenStack APIs and other support services. By default, this bridge is called br-mgmt.

The keepalived daemon is starting before NetworkManager can bring up the br-mgmt bridge and that is causing keepalived to fail. We need a way to tell systemd to wait on the network before bringing up keepalived.

Waiting on NetworkManager

There is a special systemd target, network-online.target, that is not reached until all networking is properly configured. NetworkManager comes with a handy service called NetworkManager-wait-online.service that must be complete before the network-online target can be reached:

# rpm -ql NetworkManager | grep network-online
/usr/lib/systemd/system/network-online.target.wants
/usr/lib/systemd/system/network-online.target.wants/NetworkManager-wait-online.service

Start by ensuring that the NetworkManager-wait-online service starts at boot time:

systemctl enable NetworkManager-wait-online.service

Using network-online.target

Next, we tell the keepalived service to wait on network-online.target. Bring up an editor for overriding the keepalived.service unit:

systemctl edit keepalived.service

Once the editor appears, add the following text:

[Unit]
Wants=network-online.target
After=network-online.target

Save the file in the editor and reboot the server. The keepalived service should come up successfully after NetworkManager signals that all of the network devices are online.

Learn more by reading the upstream NetworkTarget documentation.

Changes in RHEL 7 Security Technical Implementation Guide Version 1, Release 3

major@mhtx.net (Major Hayden) — Thu, 02 Nov 2017 15:00:25 +0000

The latest release of the Red Hat Enterprise Linux Security Technical Implementation Guide (STIG) was published last week. This release is Version 1, Release 3, and it contains four main changes:

V-77819 - Multifactor authentication is required for graphical logins
V-77821 - Datagram Congestion Control Protocol (DCCP) kernel module must be disabled
V-77823 - Single user mode must require user authentication
V-77825 - Address space layout randomization (ASLR) must be enabled

Deep dive

Let’s break down this list to understand what each one means.

V-77819 - Multifactor authentication is required for graphical logins

This requirement improves security for graphical logins and extends the existing requirements for multifactor authentication for logins (see V-71965, V-72417, and V-72427). The STIG recommends smartcards (since the US Government often uses CAC cards for multifactor authentication), and this is a good idea for high security systems.

I use Yubikey 4’s as smartcards in most situations and they work anywhere you have available USB slots.

V-77821 - Datagram Congestion Control Protocol (DCCP) kernel module must be disabled

DCCP is often used as a congestion control mechanism for UDP traffic, but it isn’t used that often in modern networks. There have been vulnerabilities in the past that are mitigated by disabling DCCP, so it’s a good idea to disable it unless you have a strong reason for keeping it enabled.

The ansible-hardening role has been updated to disable the DCCP kernel module by default.

V-77823 - Single user mode must require user authentication

Single user mode is often used in emergency situations where the server cannot boot properly or an issue must be repaired without a fully booted server. This mode can only be used at the server’s physical console, serial port, or via out-of-band management (DRAC, iLO, and IPMI). Allowing single-user mode access without authentication is a serious security risk.

Fortunately, every distribution supported by the ansible-hardening role already has authentication requirements for single user mode in place. The ansible-hardening role does not make any adjustments to the single user mode unit file since any untested adjustment could cause a system to have problems booting.

V-77825 - Address space layout randomization (ASLR) must be enabled

ASLR is a handy technology that makes it more difficult for attackers to guess where a particular program is storing data in memory. It’s not perfect, but it certainly raises the difficulty for an attacker. There are multiple settings for this variable and the kernel documentation for sysctl has some brief explanations for each setting (search for randomize_va_space on the page).

Every distribution supported by the ansible-hardening role is already setting kernel.randomize_va_space=2 by default, which applies randomization for the basic parts of process memory (such as shared libraries and the stack) as well as the heap. The ansible-hardening role will ensure that the default setting is maintained.

ansible-hardening is already up to date

If you’re already using the ansible-hardening role’s master branch, these changes are already in place! Try out the new updates and open a bug report if you find any problems.

Import RPM repository GPG keys from other keyservers temporarily

major@mhtx.net (Major Hayden) — Wed, 20 Sep 2017 15:24:13 +0000

I’ve been working through some patches to OpenStack-Ansible lately to optimize how we configure yum repositories in our deployments. During that work, I ran into some issues where pgp.mit.edu was returning 500 errors for some requests to retrieve GPG keys.

Ansible was returning this error:

curl: (22) The requested URL returned error: 502 Proxy Error
error: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x61E8806C: import read failed(2)

How does the rpm command know which keyserver to use? Let’s use the --showrc argument to show how it is configured:

$ rpm --showrc | grep hkp
-14: _hkp_keyserver http://pgp.mit.edu
-14: _hkp_keyserver_query %{_hkp_keyserver}:11371/pks/lookup?op=get&search=0x

How do we change this value temporarily to test a GPG key retrieval from a different server? There’s an argument for that as well: --define:

$ rpm --help | grep define
 -D, --define='MACRO EXPR' define MACRO with value EXPR

We can assemble that on the command line to set a different keyserver temporarily:

# rpm -vv --define="%_hkp_keyserver http://pool.sks-keyservers.net" --import 0x61E8806C
-- SNIP --
D: adding "63deac79abe7ad80e147d671c2ac5bd1c8b3576e" to Sha1header index.
-- SNIP --

Let’s verify that our new key is in place:

# rpm -qa | grep -i gpg-pubkey-61E8806C
gpg-pubkey-61e8806c-5581df56
# rpm -qi gpg-pubkey-61e8806c-5581df56
Name : gpg-pubkey
Version : 61e8806c
Release : 5581df56
Architecture: (none)
Install Date: Wed 20 Sep 2017 10:17:11 AM CDT
Group : Public Keys
Size : 0
License : pubkey
Signature : (none)
Source RPM : (none)
Build Date : Wed 17 Jun 2015 03:57:58 PM CDT
Build Host : localhost
Relocations : (not relocatable)
Packager : CentOS Virtualization SIG (http://wiki.centos.org/SpecialInterestGroup/Virtualization) <security@centos.org>
Summary : gpg(CentOS Virtualization SIG (http://wiki.centos.org/SpecialInterestGroup/Virtualization) <security@centos.org>)
Description :
-----BEGIN PGP PUBLIC KEY BLOCK-----
Version: rpm-4.11.3 (NSS-3)

mQENBFWB31YBCAC4dFmTzBDOcq4R1RbvQXLkyYfF+yXcsMA5kwZy7kjxnFqBoNPv
aAjFm3e5huTw2BMZW0viLGJrHZGnsXsE5iNmzom2UgCtrvcG2f65OFGlC1HZ3ajA
8ZIfdgNQkPpor61xqBCLzIsp55A7YuPNDvatk/+MqGdNv8Ug7iVmhQvI0p1bbaZR
0GuavmC5EZ/+mDlZ2kHIQOUoInHqLJaX7iw46iLRUnvJ1vATOzTnKidoFapjhzIt
i4ZSIRaalyJ4sT+oX4CoRzerNnUtIe2k9Hw6cEu4YKGCO7nnuXjMKz7Nz5GgP2Ou
zIA/fcOmQkSGcn7FoXybWJ8DqBExvkJuDljPABEBAAG0bENlbnRPUyBWaXJ0dWFs
aXphdGlvbiBTSUcgKGh0dHA6Ly93aWtpLmNlbnRvcy5vcmcvU3BlY2lhbEludGVy
ZXN0R3JvdXAvVmlydHVhbGl6YXRpb24pIDxzZWN1cml0eUBjZW50b3Mub3JnPokB
OQQTAQIAIwUCVYHfVgIbAwcLCQgHAwIBBhUIAgkKCwQWAgMBAh4BAheAAAoJEHrr
voJh6IBsRd0H/A62i5CqfftuySOCE95xMxZRw8+voWO84QS9zYvDEnzcEQpNnHyo
FNZTpKOghIDtETWxzpY2ThLixcZOTubT+6hUL1n+cuLDVMu4OVXBPoUkRy56defc
qkWR+UVwQitmlq1ngzwmqVZaB8Hf/mFZiB3B3Jr4dvVgWXRv58jcXFOPb8DdUoAc
S3u/FLvri92lCaXu08p8YSpFOfT5T55kFICeneqETNYS2E3iKLipHFOLh7EWGM5b
Wsr7o0r+KltI4Ehy/TjvNX16fa/t9p5pUs8rKyG8SZndxJCsk0MW55G9HFvQ0FmP
A6vX9WQmbP+ml7jsUxtEJ6MOGJ39jmaUvPc=
=ZzP+
-----END PGP PUBLIC KEY BLOCK-----

Success!

If you want to override the value permanently, create a ~/.rpmmacros file and add the following line to it:

%_hkp_keyserver http://pool.sks-keyservers.net

Photo credit: Wikipedia

RHEL 7 STIG v1 updates for openstack-ansible-security

major@mhtx.net (Major Hayden) — Wed, 05 Apr 2017 17:46:17 +0000

DISA’s final release of the Red Hat Enterprise Linux (RHEL) 7 Security Technical Implementation Guide (STIG) came out a few weeks ago and it has plenty of improvements and changes. The openstack-ansible-security role has already been updated with these changes.

Quite a few duplicated STIG controls were removed and a few new ones were added. Some of the controls in the pre-release were difficult to implement, especially those that changed parameters for PKI-based authentication.

The biggest challenge overall was the renumbering. The pre-release STIG used an unusual numbering convention: RHEL-07-123456. The final version used the more standardized “V” numbers, such as V-72225. This change required a substantial patch to bring the Ansible role inline with the new STIG release.

All of the role’s documentation is now updated to reflect the new numbering scheme and STIG changes. The key thing to remember is that you’ll need to use --skip-tag with the new STIG numbers if you need to skip certain tasks.

Note: These changes won’t be backported to the stable/ocata branch, so you need to use the master branch to get these changes.

Have feedback? Found a bug? Let us know!

IRC: #openstack-ansible on Freenode IRC
Bugs: LaunchPad
E-mail: openstack-dev@lists.rackspace.com with the subject line [openstack-ansible][security]

Display auditd messages with journalctl

major@mhtx.net (Major Hayden) — Thu, 05 Jan 2017 15:53:13 +0000

All systems running systemd come with a powerful tool for reviewing the system journal: journalctl. It allows you to get a quick look at the system journal while also allowing you to heavily customize your view of the log.

I logged into a server recently that was having a problem and I found that the audit logs weren’t going into syslog. That’s no problem - they’re in the system journal. The system journal was filled with tons of other messages, so I decided to limit the output only to messages from the auditd unit:

$ sudo journalctl -u auditd --boot
-- Logs begin at Thu 2015-11-05 09:20:01 CST, end at Thu 2017-01-05 09:38:49 CST. --
Jan 05 07:47:04 arsenic systemd[1]: Starting Security Auditing Service...
Jan 05 07:47:04 arsenic auditd[937]: Started dispatcher: /sbin/audispd pid: 949
Jan 05 07:47:04 arsenic audispd[949]: priority_boost_parser called with: 4
Jan 05 07:47:04 arsenic audispd[949]: max_restarts_parser called with: 10
Jan 05 07:47:04 arsenic audispd[949]: audispd initialized with q_depth=150 and 1 active plugins
Jan 05 07:47:04 arsenic augenrules[938]: /sbin/augenrules: No change
Jan 05 07:47:04 arsenic augenrules[938]: No rules
Jan 05 07:47:04 arsenic auditd[937]: Init complete, auditd 2.7 listening for events (startup state enable)
Jan 05 07:47:04 arsenic systemd[1]: Started Security Auditing Service.

This isn’t helpful. I’m seeing messages about the auditd daemon itself. I want the actual output from the audit rules.

Then I remembered: the kernel is the one that sends messages about audit rules to the system journal. Let’s just look at what’s coming from the kernel instead:

$ sudo journalctl -k --boot
-- Logs begin at Thu 2015-11-05 09:20:01 CST, end at Thu 2017-01-05 09:40:44 CST. --
Jan 05 07:46:47 arsenic kernel: Linux version 4.8.15-300.fc25.x86_64 (mockbuild@bkernel01.phx2.fedoraproject.org) (gcc version 6.2.1 20160916 (Red Hat 6.2.1-2
Jan 05 07:46:47 arsenic kernel: Command line: BOOT_IMAGE=/vmlinuz-4.8.15-300.fc25.x86_64 root=/dev/mapper/luks-e... ro rd.luks
Jan 05 07:46:47 arsenic kernel: x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers'
Jan 05 07:46:47 arsenic kernel: x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers'
Jan 05 07:46:47 arsenic kernel: x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers'
Jan 05 07:46:47 arsenic kernel: x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256

This is worse! Luckily, the system journal keeps a lot more data about what it receives than just the text of the log line. We can dig into that extra data with the verbose option:

$ sudo journalctl --boot -o verbose

After running that command, search for one of the audit log lines in the output:

_UID=0
_BOOT_ID=...
_MACHINE_ID=...
_HOSTNAME=arsenic
_TRANSPORT=audit
SYSLOG_FACILITY=4
SYSLOG_IDENTIFIER=audit
AUDIT_FIELD_HOSTNAME=?
AUDIT_FIELD_ADDR=?
AUDIT_FIELD_RES=success
_AUDIT_TYPE=1105
AUDIT_FIELD_OP=PAM:session_open
_SELINUX_CONTEXT=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
_AUDIT_LOGINUID=1000
_AUDIT_SESSION=3
AUDIT_FIELD_ACCT=root
AUDIT_FIELD_EXE=/usr/bin/sudo
AUDIT_FIELD_GRANTORS=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix
AUDIT_FIELD_TERMINAL=/dev/pts/4
_PID=2666
_SOURCE_REALTIME_TIMESTAMP=1483631103122000
_AUDIT_ID=385
MESSAGE=USER_START pid=2666 uid=0 auid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:session_open grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'

One of the identifiers we can use is _TRANSPORT=audit. Let’s pass that to journalctl and see what we get:

$ sudo journalctl --boot _TRANSPORT=audit
-- Logs begin at Thu 2015-11-05 09:20:01 CST. --
Jan 05 09:47:24 arsenic audit[3028]: USER_END pid=3028 uid=0 auid=1000 ses=3 subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 msg='op=PAM:session_close grantors=pam_keyinit,pam_limits,pam_keyinit,pam_limits,pam_systemd,pam_unix acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/4 res=success'
... more log lines snipped ...

Success! You can get live output of the audit logs by tailing the output:

sudo journalctl -af _TRANSPORT=audit

For more details on journalctl, refer to the online documentation.

augenrules fails with “rule exists” when loading rules into auditd

major@mhtx.net (Major Hayden) — Tue, 03 Jan 2017 19:01:46 +0000

When I came back from the holiday break, I found that the openstack-ansible-security role wasn’t passing tests any longer. The Ansible playbook stopped when augenrules ran to load the new audit rules. The error wasn’t terribly helpful:

/usr/sbin/augenrules: No change
Error sending add rule data request (Rule exists)
There was an error in line 5 of /etc/audit/audit.rules

A duplicated rule?

I’ve been working on lots of changes to implement the Red Hat Enterprise Linux 7 Security Technical Implementation Guide (STIG) and I assumed I put in the same rule twice with an errant copy and paste.

That wasn’t the case. I checked the input rule file in /etc/audit/rules.d/ and found that all of the rules were unique.

Is something missing?

The augenrules command works by taking files from /etc/audit/rules.d/ and joining them together into /etc/audit/audit.rules. Based on the output from augenrules, the rule file checks out fine and it determined that the existing rule doesn’t need to be updated. However, augenrules is still unable to load the new rules into auditd.

I decided to check the first several lines of /etc/audit/rules.d/ to see if line 5 had a problem:

## This file is automatically generated from /etc/audit/rules.d


-f 1
-a always,exit -F path=/usr/bin/chsh -F perm=x -F auid>=1000 -F auid!=4294967295 -k RHEL-07-030525
-a always,exit -F path=/usr/bin/chage -F perm=x -F auid>=1000 -F auid!=4294967295 -k RHEL-07-030513

Two things looked strange to me:

Line 5 is correct and it is unique
Why are lines 2 and 3 blank?

I checked another CentOS 7 server and found the following in lines 2 and 3:

-D
-b 320

The -D deletes all previously loaded rules and -b increases the buffer size for busy periods. My rules weren’t loading properly because the -D was missing! Those two lines normally come from /etc/audit/rules.d/audit.rules, but that default file was not present.

Here’s what was going wrong:

augenrules read rules from rules.d/
augenrules found that the rules in rules.d/ were already in the main audit.rules file and didn’t need to be updated
augenrules attempted to load the rules into auditd, but that failed
auditd was rejecting the rules because at least one of them (line 5) already existed in the running rule set

All of this happened because the -D wasn’t handled first before new rules were loaded.

Fixing it

I decided to add the -D line explicitly in my rules file within rules.d/ to catch those situations when the audit.rules default file is missing. The augenrules command ensures that the line appears at the top of the rules when they are loaded into auditd.

Automated security hardening with Ansible: May updates

major@mhtx.net (Major Hayden) — Fri, 27 May 2016 02:40:33 +0000

Lots of work has gone into the openstack-ansible-security Ansible role since I delivered a talk about it last month at the OpenStack Summit in Austin. Attendees asked for quite a few new features and I’ve seen quite a few bug reports (and that’s a good thing).

Here’s a list of the newest additions since the Summit:

New features

Ubuntu 16.04 LTS (Xenial) support

The role now works with Ubuntu 16.04 and its newest features, including systemd. You can use the same variables as you used with Ubuntu 14.04 and it should take the same actions. Documentation updates are mostly merged with a few straggling reviews in the queue.

CentOS 7 support

With all of the work going into the role to support Ubuntu 16.04 and systemd, CentOS 7 wasn’t a huge stretch. Many of the package names and file locations were a little different, but those are now moved out into variables files to reduce the repetition of tasks. Some of the Linux Security Module tasks needed adjustments since SELinux is a different beast than AppArmor.

Following the STIG more closely

One of the common questions I had at the summit was: “Can I use this thing on my non-OpenStack environments?” You definitely can, but many of the configurations were tweaked to avoid causing problems with OpenStack environments. Some users asked if the configurations could be made more generic so that they followed the STIG more closely. This would reduce some compliance headaches and allow more people to use the role.

So far, I’ve been making some of these adjustments to fix more things rather than simply checking them. That should make it easier to get closer to the STIG’s requirements.

Another proposed idea is to create vars files that meet different criteria. For example, one vars file might be the ultra-secure, follow-the-STIG-to-the-letter configuration. This would be good for users that already know they want to apply the STIG’s requirements fully. There could be another vars file that would apply most of the STIG’s requirements, but it would steer clear of changing anything that could disrupt a production OpenStack environment.

The future

Here are a subset of the future plans and ideas:

Better reporting for users who need to feed data into vulnerability management applications or SIEMs for compliance checks
Better testing, possibly with customized OpenSCAP XCCDF files
Cross-referenced controls to other hardening guides, such as CIS Benchmarks

If you have any other ideas, feel free to stop by #openstack-ansible or #openstack-security on Freenode. You can find me there as mhayden and I would really enjoy hearing about your use cases!

Photo credit: Mikecogh

Updating Dell PowerEdge BIOS from Linux

major@mhtx.net (Major Hayden) — Mon, 18 Jan 2016 20:53:38 +0000

Updating Dell PowerEdge firmware from Linux is quite easy, but it isn’t documented very well. I ended up with a set of PowerEdge R710’s at work for a lab environment and the BIOS versions were different on each server.

Downloading the latest firmware

Start by heading over to Dell’s support site and enter your system’s service tag. You can use lshw to find your service tag:

# lshw | head
lab05
 description: Rack Mount Chassis
 product: PowerEdge R710 ()
 vendor: Dell Inc.
 serial: [service tag should be here]
 width: 64 bits
 capabilities: smbios-2.6 dmi-2.6 vsyscall32
 configuration: boot=normal chassis=rackmount uuid=44454C4C-3700-104A-8052-B2C04F564831
 *-core
 description: Motherboard

After entering the service tag, follow these steps:

Click Drivers & downloads on the left
Click Change OS at the top right and choose Red Hat Enterprise Linux 7
Click the BIOS dropdown in the list
Click Other file formats available
Look for the file ending in BIN and click Download file underneath it

Copy that file to your server that needs a BIOS update.

Installing firmware update tools

Start by getting the right packages installed. I’ll cover the CentOS/RHEL and Ubuntu methods here. At the moment, Fedora doesn’t build kernels with the dell_rbu module enabled, but there’s a discussion about getting that fixed.

For CentOS, you’ll need to get the Dell Linux repository configured first:

wget http://linux.dell.com/repo/hardware/latest/bootstrap.cgi
sh bootstrap.cgi
yum -y install firmware-addon-dell

For Ubuntu, the package is in the upstream repositories already:

apt-get -y install firmware-addon-dell

Extract and flash the BIOS header

Dell packages up a BIOS header (the actual firmware blob that needs to be flashed) within the BIN file you downloaded earlier. The latest version of the BIOS for my R710 is 6.4.0, so my file is called R710_BIOS_4HKX2_LN_6.4.0.BIN. Let’s start by extracting the header file:

bash R710_BIOS_4HKX2_LN_6.4.0.BIN --extract bios

You should now have a directory in your current directory called bios. The header file is within bios/payload/ and you’ll use that to flash the BIOS:

# modprobe dell_rbu
# dellBiosUpdate-compat --hdr bios/payload/R710-060400.hdr --update
Supported RBU type for this system: (MONOLITHIC, PACKET)
Using RBU v2 driver. Initializing Driver.
Setting RBU type in v2 driver to: PACKET
writing (4096) to file: /sys/devices/platform/dell_rbu/packet_size
Writing RBU data (4096bytes/dot): ...........................
Done writing packet data.
Activate CMOS bit to notify BIOS that update is ready on next boot.
Update staged sucessfully. BIOS update will occur on next reboot.

It’s now time to reboot! If you watch the console via iDRAC, you’ll see a 3-4 minute delay on the next reboot while the staged BIOS image is flashed. When the server boots, use lshw to verify that the BIOS version has been updated.

Photo Credit: vaxomatic via Compfight cc

Chronicles of SELinux: Dealing with web content in unusual directories

major@mhtx.net (Major Hayden) — Thu, 10 Sep 2015 13:40:35 +0000

I’ve decided to start a series of posts called “Chronicles of SELinux” where I hope to educate more users on how to handle SELinux denials with finesse rather than simply disabling it entirely. To kick things off, I’ll be talking about dealing with web content in the first post.

First steps

If you’d like to follow along, simply hop onto a system running Fedora 21 (or later), CentOS 7 or Red Hat Enterprise Linux 7. We need SELinux in enforcing mode on the host, so be sure to check the status with getenforce. Depending on what getenforce returns, you’ll need to make adjustments:

Enforcing: No adjustments needed - you’re all set!
Permissive: Run setenforce 1 and adjust SELinux configuration file (see below)
Disabled: Adjust the SELinux configuration file and reboot (see below)

To enable enforcing mode in the SELinux configuration file, edit /etc/selinux/config and ensure your SELINUX line has enforcing:

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=enforcing

If getenforce returned Disabled earlier, you will need to reboot to get SELinux working. Also be sure that the selinux-policy-targeted package is installed and run fixfiles onboot -B to relabel the system on reboot (thanks to immanetize for the comment).

Let’s install httpd and create a developer user:

# For Fedora
dnf -y install httpd
# For CentOS/RHEL
yum -y install httpd

useradd developer
systemctl enable httpd
systemctl start httpd

On to the guide!

Hosting content in an unique directory

On Red Hat-based systems, httpd expects to find its content in /var/www/html, but some system administrators prefer to have content stored elsewhere on the system. It could be on a SAN or other remote storage, but it could also just be in a different directory to make things easier for the business.

Let’s consider a situation where the web content is hosted from /web/. We can create the directory:

[root@fedora22 ~]# mkdir -v /web
mkdir: created directory '/web'

We can edit /etc/httpd/conf/httpd.conf and set our new DocumentRoot:

#
# DocumentRoot: The directory out of which you will serve your
# documents. By default, all requests are taken from this directory, but
# symbolic links and aliases may be used to point to other locations.
#
DocumentRoot "/web"

Let’s reload the httpd configuration:

systemctl reload httpd

And now we can add some amazing web content:

 /web/index.html

It’s time to test our web server:

# curl -i localhost/index.html
HTTP/1.1 403 Forbidden
Date: Thu, 10 Sep 2015 12:54:19 GMT
Server: Apache/2.4.16 (Fedora)
Content-Length: 219
Content-Type: text/html; charset=iso-8859-1

Oh, come on. What’s with this 403 error?

Investigating the 403

The first step for any situation like this is to review some logs. Let’s check the logs for httpd:

[Thu Sep 10 12:55:04.541789 2015] [core:error] [pid 16597] (13)Permission denied: [client ::1:49860] AH00035: access to /index.html denied (filesystem path '/web/index.html') because search permissions are missing on a component of the path

Search permissions are missing? What? Let’s check the permissions on our web directory:

# ls -al /web
total 12
drwxr-xr-x. 2 root root 4096 Sep 10 12:53 .
dr-xr-xr-x. 19 root root 4096 Sep 10 12:51 ..
-rw-r--r--. 1 root root 21 Sep 10 12:54 index.html

The httpd user has the ability to get into the directory (o+x is set on /web/) and the httpd user can read the file (o+r is set on /web/index.html). Let’s check the system journal just in case:

# journalctl -n 1 | tail
-- Logs begin at Thu 2015-09-10 12:31:37 UTC, end at Thu 2015-09-10 12:55:04 UTC. --
Sep 10 12:55:04 fedora22 audit[16597]: <audit-1400> avc: denied { getattr } for pid=16597 comm="httpd" path="/web/index.html" dev="xvda1" ino=524290 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:default_t:s0 tclass=file permissive=0

That’s quite a long log line. Let’s break it into pieces:

avc: denied { getattr } for pid=16597 comm="httpd": httpd tried to do something and was denied
path="/web/index.html" dev="xvda1" ino=524290: path to the file (index.html) involved in the denial
scontext=system_u:system_r:httpd_t:s0: the SELinux context of the httpd process
tcontext=unconfined_u:object_r:default_t:s0: the SELinux contect that is actually applied to our index.html
tclass=file: the denial came from accessing a file (index.html)
permissive=0: we’re in enforcing mode, not permissive mode

Long story short, when httpd tried to access our /web/index.html file, the httpd process was labeled with httpd_t, but the kernel found that the HTML file was labeled with default_t. The httpd process (labeled with httpd_t) isn’t allowed to read files that are labeled as default_t, so the access is denied.

Fixing it the right way

Since we know what SELinux expects for this file (from the log line in the journal), we can apply the right context and re-test. The chcon command has a handy argument that allows you to reference a file or directory, and apply the contexts from there. Since we know that /var/www/html has the right contexts already, we can use it as a reference:

# chcon -v -R --reference=/var/www/html /web
changing security context of '/web/index.html'
changing security context of '/web'

Now we see some different contexts on /web:

# ls -alZ /web/
total 12
drwxr-xr-x. 2 root root system_u:object_r:httpd_sys_content_t:s0 4096 Sep 10 13:19 .
dr-xr-xr-x. 19 root root system_u:object_r:root_t:s0 4096 Sep 10 13:19 ..
-rw-r--r--. 1 root root system_u:object_r:httpd_sys_content_t:s0 21 Sep 10 13:19 index.html

Let’s test again:

# curl -I localhost/index.html
HTTP/1.1 403 Forbidden
Date: Thu, 10 Sep 2015 13:21:22 GMT
Server: Apache/2.4.16 (Fedora)
Content-Type: text/html; charset=iso-8859-1

Darn! What’s in the httpd logs?

[Thu Sep 10 13:21:22.267719 2015] [authz_core:error] [pid 16593] [client ::1:49861] AH01630: client denied by server configuration: /web/index.html

Ah, we cleared the SELinux problem but now httpd is upset. Just below the DocumentRoot line that we edited earlier, look for two Directory blocks. Change /var/www/ and /var/www/html to /web in those blocks. Reload the httpd configuration and try once more:

# systemctl reload httpd
# curl -I localhost/index.html
HTTP/1.1 200 OK
Date: Thu, 10 Sep 2015 13:25:16 GMT
Server: Apache/2.4.16 (Fedora)
Last-Modified: Thu, 10 Sep 2015 13:19:47 GMT
ETag: "15-51f6474064d50"
Accept-Ranges: bytes
Content-Length: 21
Content-Type: text/html; charset=UTF-8

Success!

Long term fix

The chcon method is good for fixing one-off issues and for testing, but we need a good long term fix. SELinux has some file contexts already configured for certain directories, but not for our custom web directory. You can examine the defaults here:

# semanage fcontext -l | grep ^/var/www/html
/var/www/html(/.*)?/sites/default/files(/.*)? all files system_u:object_r:httpd_sys_rw_content_t:s0
/var/www/html(/.*)?/sites/default/settings\.php regular file system_u:object_r:httpd_sys_rw_content_t:s0
/var/www/html(/.*)?/uploads(/.*)? all files system_u:object_r:httpd_sys_rw_content_t:s0
/var/www/html(/.*)?/wp-content(/.*)? all files system_u:object_r:httpd_sys_rw_content_t:s0
/var/www/html/[^/]*/cgi-bin(/.*)? all files system_u:object_r:httpd_sys_script_exec_t:s0
/var/www/html/cgi/munin.* all files system_u:object_r:munin_script_exec_t:s0
/var/www/html/configuration\.php all files system_u:object_r:httpd_sys_rw_content_t:s0
/var/www/html/munin(/.*)? all files system_u:object_r:munin_content_t:s0
/var/www/html/munin/cgi(/.*)? all files system_u:object_r:munin_script_exec_t:s0
/var/www/html/owncloud/data(/.*)? all files system_u:object_r:httpd_sys_rw_content_t:s0

SELinux’s tools have a concept of equivalency. This allows you to say that one directory is equivalent to another one in the long term. We already used chcon to apply contexts with a reference to a directory with valid contexts, but this equivalency concept gives us a longer term fix. Here’s the command to use:

semanage fcontext --add --equal /var/www /web

If we break this down, we’re saying we want to add a new file context where /web is equal to /var/www. This means we want the same SELinux contexts applied in the same places and want them treated equally. After running the semanage command, let’s make an index2.html file to test:

 /web/index2.html
# curl -I localhost/index2.html
HTTP/1.1 200 OK
Date: Thu, 10 Sep 2015 13:35:24 GMT
Server: Apache/2.4.16 (Fedora)
Last-Modified: Thu, 10 Sep 2015 13:34:11 GMT
ETag: "15-51f64a78266c8"
Accept-Ranges: bytes
Content-Length: 21
Content-Type: text/html; charset=UTF-8

Great! We didn’t have to use chcon this time around because we configured /web as an equivalent directory to /var/www. Let’s double check the contexts:

# ls -alZ /web
total 16
drwxr-xr-x. 2 root root unconfined_u:object_r:httpd_sys_content_t:s0 4096 Sep 10 13:34 .
dr-xr-xr-x. 19 root root system_u:object_r:root_t:s0 4096 Sep 10 13:33 ..
-rw-r--r--. 1 root root unconfined_u:object_r:httpd_sys_content_t:s0 21 Sep 10 13:34 index2.html
-rw-r--r--. 1 root root unconfined_u:object_r:httpd_sys_content_t:s0 21 Sep 10 13:33 index.html

Perfect! We now have all of the security benefits of SELinux in a completely custom web directory.

Understanding systemd’s predictable network device names

major@mhtx.net (Major Hayden) — Fri, 21 Aug 2015 21:15:36 +0000

I talked a bit about systemd’s network device name in my earlier post about [systemd-networkd and bonding][2] and I received some questions about how systemd rolls through the possible names of network devices to choose the final name. These predictable network device names threw me a curveball last summer when I couldn’t figure out how the names were constructed.

Let’s walk through this process.

What’s in a name?

Back in the systemd-networkd bonding post, I dug into a dual port Intel network card that showed up in a hotplug slot:

# udevadm info -e | grep -A 9 ^P.*eth0
P: /devices/pci0000:00/0000:00:03.2/0000:08:00.0/net/eth0
E: DEVPATH=/devices/pci0000:00/0000:00:03.2/0000:08:00.0/net/eth0
E: ID_BUS=pci
E: ID_MODEL_FROM_DATABASE=82599ES 10-Gigabit SFI/SFP+ Network Connection (Ethernet OCP Server Adapter X520-2)
E: ID_MODEL_ID=0x10fb
E: ID_NET_DRIVER=ixgbe
E: ID_NET_LINK_FILE=/usr/lib/systemd/network/99-default.link
E: ID_NET_NAME_MAC=enxa0369f2cec90
E: ID_NET_NAME_PATH=enp8s0f0
E: ID_NET_NAME_SLOT=ens9f0

This udev database dump shows that it came up with a few different names for the network interface:

ID_NET_NAME_MAC=enxa0369f2cec90
ID_NET_NAME_PATH=enp8s0f0
ID_NET_NAME_SLOT=ens9f0

Where do these names come from? We can dig into systemd’s source code to figure out the origin of the names and which one is selected as the final choice.

Down the udev rabbit hole

Let’s take a look at src/udev/udev-builtin-net_id.c:

/*
 * Predictable network interface device names based on:
 * - firmware/bios-provided index numbers for on-board devices
 * - firmware-provided pci-express hotplug slot index number
 * - physical/geographical location of the hardware
 * - the interface's MAC address
 *
 * http://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames
 *
 * Two character prefixes based on the type of interface:
 * en -- ethernet
 * sl -- serial line IP (slip)
 * wl -- wlan
 * ww -- wwan
 *
 * Type of names:
 * b<number> -- BCMA bus core number
 * ccw<name> -- CCW bus group name
 * o<index>[d<dev_port>] -- on-board device index number
 * s<slot>[f<function>][d<dev_port>] -- hotplug slot index number
 * x<MAC> -- MAC address
 * [P<domain>]p<bus>s<slot>[f<function>][d<dev_port>]
 * -- PCI geographical location
 * [P<domain>]p<bus>s<slot>[f<function>][u<port>][..][c<config>][i<interface>]
 * -- USB port number chain

So here’s where our names actually begin. Ethernet cards will always start with en, but they might be followed by a p (for PCI slots), a s (for hotplug PCI-E slots), and o (for onboard cards). Scroll down just a bit more for some examples starting at line 56.

Real-world examples

We already looked at the hotplug slot naming from Rackspace’s OnMetal servers. They show up as ens9f0 and ens9f1. That means they’re on a hotplug slot which happens to be slot 9. The function indexes are 0 and 1 (for both ports on the Intel 82599ES).

Linux firewall with a dual-port PCI card

Here’s an example of my Linux firewall at home. It’s a Dell Optiplex 3020 with an Intel I350-T2 (dual port):

# udevadm info -e | grep -A 10 ^P.*enp1s0f1
P: /devices/pci0000:00/0000:00:01.0/0000:01:00.1/net/enp1s0f1
E: DEVPATH=/devices/pci0000:00/0000:00:01.0/0000:01:00.1/net/enp1s0f1
E: ID_BUS=pci
E: ID_MODEL_FROM_DATABASE=I350 Gigabit Network Connection (Ethernet Server Adapter I350-T2)
E: ID_MODEL_ID=0x1521
E: ID_NET_DRIVER=igb
E: ID_NET_LINK_FILE=/usr/lib/systemd/network/99-default.link
E: ID_NET_NAME=enp1s0f1
E: ID_NET_NAME_MAC=enxa0369f6e5227
E: ID_NET_NAME_PATH=enp1s0f1
E: ID_OUI_FROM_DATABASE=Intel Corporate

And the output from lspci:

# lspci -s 01:00
01:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)

This card happens to sit on PCI bus 1 (enp1), slot 0 (s0). Since it’s a dual-port card, it has two function indexes (f0 and f1). That leaves me with two predictable names: enp1s0f1 and enp1s0f0.

1U server with four ethernet ports

Let’s grab another example. Here’s a SuperMicro 1U X9SCA server with four onboard PCI ethernet cards:

# udevadm info -e | grep -A 10 ^P.*enp2s0
P: /devices/pci0000:00/0000:00:1c.4/0000:02:00.0/net/enp2s0
E: DEVPATH=/devices/pci0000:00/0000:00:1c.4/0000:02:00.0/net/enp2s0
E: ID_BUS=pci
E: ID_MODEL_FROM_DATABASE=82574L Gigabit Network Connection
E: ID_MODEL_ID=0x10d3
E: ID_NET_DRIVER=e1000e
E: ID_NET_LINK_FILE=/usr/lib/systemd/network/99-default.link
E: ID_NET_NAME=enp2s0
E: ID_NET_NAME_MAC=enx00259025963a
E: ID_NET_NAME_PATH=enp2s0
E: ID_OUI_FROM_DATABASE=Super Micro Computer, Inc.

And here’s all four ports in lspci:

# for i in `seq 2 5`; do lspci -s 0${i}:; done
02:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
03:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
04:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
05:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

These are interesting because they’re not all on the same PCI bus. They sit on buses 2-5 in slot 0. There are no function indexes here, so they’re named enp2s0 through enp5s0. These aren’t true onboard cards, so they’re named based on their locations.

Storage server with onboard ethernet

Here’s an example of a server with a true inboard ethernet card:

$ udevadm info -e | grep -A 11 ^P.*eno1
P: /devices/pci0000:00/0000:00:19.0/net/eno1
E: DEVPATH=/devices/pci0000:00/0000:00:19.0/net/eno1
E: ID_BUS=pci
E: ID_MODEL_FROM_DATABASE=Ethernet Connection I217-V
E: ID_MODEL_ID=0x153b
E: ID_NET_DRIVER=e1000e
E: ID_NET_LABEL_ONBOARD=en Onboard LAN
E: ID_NET_LINK_FILE=/usr/lib/systemd/network/99-default.link
E: ID_NET_NAME_MAC=enxe03f49b159c0
E: ID_NET_NAME_ONBOARD=eno1
E: ID_NET_NAME_PATH=enp0s25
E: ID_OUI_FROM_DATABASE=ASUSTek COMPUTER INC.

And the lspci output:

$ lspci -s 00:19.0
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 05)

This card has a new name showing up in udev: ID_NET_NAME_ONBOARD. The systemd udev code has some special handling for onboard cards because they usually sit on the main bus. The naming can get a bit ugly because that 19 would need to be converted into hex for the name.

If systemd didn’t handle onboard cards differently, this card might be named something ugly like enp0s13 (since 19 in decimal becomes 13 in hex). That’s really confusing.

Picking the final name

As we’ve seen above, udev makes a big list of names in the udev database. However, there can only be one name in the OS when you try to use the network card.

Let’s wander back into the code. this time we’re going to take a look in src/udev/net/link-config.c starting at around line 403:

name_policy) {
 NamePolicy *policy;

 for (policy = config->name_policy;
 !new_name && *policy != _NAMEPOLICY_INVALID; policy++) {
 switch (*policy) {
 case NAMEPOLICY_KERNEL:
 respect_predictable = true;
 break;
 case NAMEPOLICY_DATABASE:
 new_name = udev_device_get_property_value(device, "ID_NET_NAME_FROM_DATABASE");
 break;
 case NAMEPOLICY_ONBOARD:
 new_name = udev_device_get_property_value(device, "ID_NET_NAME_ONBOARD");
 break;
 case NAMEPOLICY_SLOT:
 new_name = udev_device_get_property_value(device, "ID_NET_NAME_SLOT");
 break;
 case NAMEPOLICY_PATH:
 new_name = udev_device_get_property_value(device, "ID_NET_NAME_PATH");
 break;
 case NAMEPOLICY_MAC:
 new_name = udev_device_get_property_value(device, "ID_NET_NAME_MAC");
 break;
 default:
 break;
 }
 }
}

If we look at the overall case statement, you can see that the first match is the one that takes precedence. Working from top to bottom, udev takes the first match of:

ID_NET_NAME_FROM_DATABASE
ID_NET_NAME_ONBOARD
ID_NET_NAME_SLOT
ID_NET_NAME_PATH
ID_NET_NAME_MAC

If we go back to our OnMetal example way at the top of the post, we can follow the logic. The udev database contained the following:

E: ID_NET_NAME_MAC=enxa0369f2cec90
E: ID_NET_NAME_PATH=enp8s0f0
E: ID_NET_NAME_SLOT=ens9f0

The udev daemon would start with ID_NET_NAME_FROM_DATABASE, but that doesn’t exist for this card. Next, it would move to ID_NET_NAME_ONBOARD, but that’s not present. Next comes ID_NET_NAME_SLOT, and we have a match! The ID_NET_NAME_SLOT entry has ens9f0 and that’s the final name for the network device.

This loop also handles some special cases. The first check is to see if someone requested for udev to not use predictable naming. If the bootloader configuration contains net.ifnames=0, predictable naming logic is skipped.

Another special case is ID_NET_NAME_FROM_DATABASE. Those ports come from udev’s internal hardware database. That file only has one item at the moment and it’s for a particular Dell iDRAC network interface.

Perplexed by hex

If the PCI slot numbers don’t seem to line up, be sure to read my post from last summer. I ran into a peculiar Dell server with a dual port Intel card on PCI bus 42. The interface ended up with a name of enp66s0f0 and I was stumped.

The name enp66s0f0 seems to say that we have a card on PCI bus 66, in slot 0, with multiple function index numbers (for multiple ports). However, systemd does a conversion of PCI slot numbers into hex. That means that decimal 66 becomes 42 in hex.

Most servers won’t be this complicated, but it’s key to remember the hex conversion.

Feedback

Are these systemd-related posts interesting? Let me know. I’m a huge fan of systemd and I enjoy writing about it.

Photo credit: University of Michigan Library

Research Paper: Securing Linux Containers

major@mhtx.net (Major Hayden) — Fri, 14 Aug 2015 20:45:50 +0000

It seems like there’s a new way to run containers every week. The advantages and drawbacks of each approach are argued about on mailing lists, in IRC channels, and in person, around the world. However, the largest amount of confusion seems to be around security.

Launching secure containers

I’ve written about launching secure containers on this blog many times before:

However, my goal this time around was to do something more comprehensive and slightly more formal. After getting my GSEC and GCUX certifications from SANS/GIAC, there was an option to enhance the certification to a gold status by writing a peer-reviewed research paper on a topic related to the exam. It was a great experience to go through the review process and get feedback on the technical material as well as the structure of the paper itself.

The paper

Without further ado, here are links to the Securing Linux Containers paper:

PDF version without watermarks
PDF version from SANS (has some watermarks and SANS/GIAC extra pages)

The paper is written for readers who have some level of familiarity with Linux and some virtualization technologies. It’s a useful paper even for people who haven’t worked with containers.

It starts with an overview of Linux containers and how they differ from other types of virtualization, such as KVM or Xen. From there, it covers how to secure the host system underneath the containers and how to provide security within the containers themselves. There’s also a section on how to start a simple container on CentOS 7 and inspect the security controls inside and outside the container.

Licensing

I’m also very proud to announce that the paper is licensed under the Creative Commons Attribution-ShareAlike 4.0 International License (CC-BY-SA). You are free to quote it as much as you like (even for commercial purposes), but I’d ask that you maintain the same license and attribute me as the author.

Thank you

This paper wouldn’t have been possible without some serious help from these awesome people:

Richard Carbone was my advisor from SANS and he helped tremendously
Dan Walsh reviewed the content and gave me several pointers on topics to add and adjust
Paul Voccio, Antony Messerli, and Brad McConnell from Rackspace also provided feedback
My mother, Neta Greene, is the best educator I know and she fueled my interest in writing and sharing with others

Feedback

Please let me know if you spot any errors or areas that need clarification. This is one of my favorite topics and I enjoy talking about it. Find me on Freenode IRC as mhayden and I’ll be glad to talk more there.

Automated testing for Ansible CIS playbook on RHEL/CentOS 6

major@mhtx.net (Major Hayden) — Wed, 05 Aug 2015 13:13:52 +0000

I started working on the Ansible CIS playbook for CentOS and RHEL 6 back in 2014 and I’ve made a few changes to increase quality and make it easier to use.

First off, the role itself is no longer a submodule. You can now just clone the repository and get rolling. This should reduce the time it takes to get started.

Also, all pull requests to the repository now go through integration testing at Rackspace. Each pull request goes through the gauntlet:

Syntax check on Travis-CI
Travis-CI builds a server at Rackspace
The entire Ansible playbook runs on the Rackspace Cloud Server
Results are sent back to GitHub

The testing process usually takes under five minutes.

Stay tuned: Updates are coming for RHEL and CentOS 7. ;)

Allow new windows to steal focus in GNOME 3

major@mhtx.net (Major Hayden) — Mon, 06 Jul 2015 12:36:05 +0000

GNOME 3 generally works well for me but it has some quirks. One of those quirks is that new windows don’t actually pop up on the screen with focus as they do in Windows and OS X. When opening a new window, you get a “[Windowname] is ready” notification:

My preference is for new windows to pop in front and steal focus. I can see why that’s not the default since it might cause you to type something in another window where you weren’t expecting to. Fortunately, you can enable what GNOME calls strict window focus with a quick trip to dconf-editor.

Installing dconf-editor is easy:

# RHEL/CentOS 7 and Fedora 21
yum -y install dconf-editor
# Fedora 22
dnf -y install dconf-editor

Open dconf-editor and navigate to org -> gnome -> desktop -> wm -> preferences.

Once you’re there, look for focus-new-windows. The default setting is smart which will keep new windows in the background and alert you via a notification. If you click on smart, a drop down will appear and you can select strict. That will enable functionality similar to OS X and Windows where new windows will pop up in the front and steal your focus.

The new setting takes effect immediately and there’s no need to logout or close and reopen windows.

UPDATE: If you’d like to avoid installing dconf-editor, use Alexander’s suggestion below and simply run:

gsettings set org.gnome.desktop.wm.preferences focus-new-windows 'strict'

Improving LXC template security

major@mhtx.net (Major Hayden) — Thu, 18 Jun 2015 19:52:11 +0000

I’ve been getting involved with the Fedora Security Team lately and we’re working as a group to crush security bugs that affect Fedora, CentOS (via EPEL) and Red Hat Enterprise Linux (via EPEL). During some of this work, I stumbled upon a group of Red Hat Bugzilla tickets talking about LXC template security.

The gist of the problem is that there’s a wide variance in how users and user credentials are handled by the different LXC templates. An inventory of the current situation revealed some horrifying problems with many OS templates.

Many of the templates set an awful default root password, like rooter, toor, or root. Some of the others create a regular user with sudo privileges and give it a default, predictable password unless the user specifies otherwise.

There are some bright spots, though. Fedora and CentOS templates will accept a root password from the user during the build and set a randomized password for the root user if a password isn’t specified. Ubuntu Cloud takes another approach by locking out the root user and requiring cloud-init configuration data to configure the root account.

I kicked off a mailing list thread and wrote a terrible pull request to get things underway. Stéphane Graber requested that all templates use a shared script to handle users and credentials via standardized environment variables and command line arguments. In addition, all passwords for users (regular or root) should be empty with password-less logins disabled. Those are some reasonable requests and I’m working on a shell script that’s easy to import into LXC templates.

There’s also a push to remove sshd from all LXC templates by default, but I’m hoping to keep that one tabled until the credentials issue is solved.

If you’d like to help out with the effort, let me know! I’ll probably get some code up onto Github soon and as for comments.

Keep old kernels with yum and dnf

major@mhtx.net (Major Hayden) — Mon, 18 May 2015 14:22:56 +0000

When you upgrade packages on Red Hat, CentOS and Fedora systems, the newer package replaces the older package. That means that files managed by RPM from the old package are removed and replaced with files from the newer package.

There’s one exception here: kernel packages.

Upgrading a kernel package with yum and dnf leaves the older kernel package on the system just in case you need it again. This is handy if the new kernel introduces a bug on your system or if you need to work through a compile of a custom kernel module.

However, yum and dnf will clean up older kernels once you have more than three. The oldest kernel will be removed from the system and the newest three will remain. In some situations, you may want more than three to stay on your system.

To change the setting, simply open up /etc/yum.conf or /etc/dnf/dnf.conf in your favorite text editor. Look for this line:

installonly_limit=3

To keep five kernels, simply replace the 3 with a 5. If you’d like to keep every old kernel on the system forever, just change the 3 to a 0. A zero means you never want “installonly” packages (like kernels) to ever be removed from your system.

Creating a bridge for virtual machines using systemd-networkd

major@mhtx.net (Major Hayden) — Thu, 26 Mar 2015 13:17:08 +0000

There are plenty of guides out there for making ethernet bridges in Linux to support virtual machines using built-in network scripts or NetworkManager. I decided to try my hand with creating a bridge using only systemd-networkd and it was surprisingly easy.

First off, you’ll need a version of systemd with networkd support. Fedora 20 and 21 will work just fine. RHEL/CentOS 7 and Arch Linux should also work. Much of the networkd support has been in systemd for quite a while, but if you’re looking for fancier network settings, like bonding, you’ll want at least systemd 216.

Getting our daemons in order

Before we get started, ensure that systemd-networkd will run on a reboot and NetworkManager is disabled. We also need to make a config file director for systemd-networkd if it doesn’t exist already. In addition, let’s enable the caching resolver and make a symlink to systemd’s resolv.conf:

systemctl enable systemd-networkd
systemctl disable NetworkManager
systemctl enable systemd-resolved
ln -sf /run/systemd/resolve/resolv.conf /etc/resolv.conf
mkdir /etc/systemd/network

Configure the physical network adapter

In my case, the network adapter connected to my external network is enp4s0 but yours will vary. Run ip addr to get a list of your network cards. Let’s create /etc/systemd/network/uplink.network and put the following in it:

[Match]
Name=enp4s0

[Network]
Bridge=br0

I’m telling systemd to look for a device called enp4s0 and then add it to a bridge called br0 that we haven’t configured yet. Be sure to change enp4s0 to match your ethernet card.

Make the bridge

We need to tell systemd about our new bridge network device and we also need to specify the IP configuration for it. We start by creating /etc/systemd/network/br0.netdev to specify the device:

[NetDev]
Name=br0
Kind=bridge

This file is fairly self-explanatory. We’re telling systemd that we want a device called br0 that functions as an ethernet bridge. Now create /etc/systemd/network/br0.network to specify the IP configuration for the br0 interface:

[Match]
Name=br0

[Network]
DNS=192.168.250.1
Address=192.168.250.33/24
Gateway=192.168.250.1

This file tells systemd that we want to apply a simple static network configuration to br0 with a single IPv4 address. If you want to add additional DNS servers or IPv4/IPv6 addresses, just add more DNS= and Address lines right below the ones you see above. Yes, it’s just that easy.

Let’s do this

Some folks are brave enough to stop NetworkManager and start all of the systemd services here but I prefer to reboot so that everything comes up cleanly. That will also allow you to verify that future reboots will cause the server to come back online with the right configuration. After the reboot, run networkctl and you’ll get something like this (with color):

Here’s what’s in the screenshot:

IDX LINK TYPE OPERATIONAL SETUP
 1 lo loopback carrier unmanaged
 2 enp2s0 ether off unmanaged
 3 enp3s0 ether off unmanaged
 4 enp4s0 ether degraded configured
 5 enp5s0 ether off unmanaged
 6 br0 ether routable configured
 7 virbr0 ether no-carrier unmanaged

7 links listed.

My ethernet card has four ports and only enp4s0 is in use. It has a degraded status because there is no IP address assigned to enp4s0. You can ignore that for now but it would be nice to see this made more clear in a future systemd release.

Look at br0 and you’ll notice that it’s configured and routable. That’s the best status you can get for an interface. You’ll also see that my other ethernet devices are in the unmanaged state. I could easily add more .network files to /etc/systemd/network to configure those interfaces later.

Install sysstat on Fedora 21

major@mhtx.net (Major Hayden) — Fri, 12 Dec 2014 17:55:57 +0000

One of the first tools I learned about after working with Red Hat was sysstat. It can write down historical records about your server at regular intervals. This can help you diagnose CPU usage, RAM usage, or network usage problems. In addition, sysstat also provides some handy command line utilities like vmstat, iostat, and pidstat that give you a live view of what your system is doing.

On Debian-based systems (including Ubuntu), you install the sysstat package and enable it with a quick edit to /etc/default/sysstat and the cron job takes it from there. CentOS and Fedora systems call the collector process using a cron job in /etc/cron.d and it’s enabled by default.

Fedora 21 comes with sysstat 11 and there are now systemd unit files to control the collection and management of stats. You can find the unit files by listing the files in the sysstat RPM:

$ rpm -ql sysstat | grep systemd
/usr/lib/systemd/system/sysstat-collect.service
/usr/lib/systemd/system/sysstat-collect.timer
/usr/lib/systemd/system/sysstat-summary.service
/usr/lib/systemd/system/sysstat-summary.timer
/usr/lib/systemd/system/sysstat.service

These services and timers aren’t enabled by default in Fedora 21. If you run sar after installing sysstat, you’ll see something like this:

# sar
Cannot open /var/log/sa/sa12: No such file or directory
Please check if data collecting is enabled

All you need to do is enable and start the main sysstat service:

systemctl enable sysstat
systemctl start sysstat

From there, systemd will automatically call for collection and management of the statistics using its internal timers. Opening up /usr/lib/systemd/system/sysstat-collect.timer reveals the following:

# /usr/lib/systemd/system/sysstat-collect.timer
# (C) 2014 Tomasz Torcz <tomek@pipebreaker.pl>
#
# sysstat-11.0.0 systemd unit file:
# Activates activity collector every 10 minutes

[Unit]
Description=Run system activity accounting tool every 10 minutes

[Timer]
OnCalendar=*:00/10

[Install]
WantedBy=sysstat.service

The timer unit file ensures that the sysstat-collect.service is called every 10 minutes based on the real time provided by the system clock. (There are other options to set timers based on relative time of when the server booted or when a user logged into the system). The familiar sa1 command appears in /usr/lib/systemd/system/sysstat-collect.service:

# /usr/lib/systemd/system/sysstat-collect.service
# (C) 2014 Tomasz Torcz <tomek@pipebreaker.pl>
#
# sysstat-11.0.0 systemd unit file:
# Collects system activity data
# Activated by sysstat-collect.timer unit

[Unit]
Description=system activity accounting tool
Documentation=man:sa1(8)

[Service]
Type=oneshot
User=root
ExecStart=/usr/lib64/sa/sa1 1 1

httpry 0.1.8 available for RHEL and CentOS 7

major@mhtx.net (Major Hayden) — Wed, 13 Aug 2014 13:20:28 +0000

Red Hat Enterprise Linux and CentOS 7 users can now install httpry 0.1.8 in EPEL 7 Beta. The new httpry version is also available for RHEL/CentOS 6 and supported Fedora versions (19, 20, 21 branched, and rawhide).

Configuring EPEL on a RHEL/CentOS server is easy. Follow the instructions on EPEL’s site and install the epel-release RPM that matches your OS release version.

If you haven’t used httpry before, check the output on Jason Bittel’s site. It’s a handy way to watch almost any type of HTTP server and see the traffic in an easier to read (and easier to grep) format.

Unexpected predictable network naming with systemd

major@mhtx.net (Major Hayden) — Wed, 06 Aug 2014 21:09:34 +0000

While using a Dell R720 at work today, we stumbled upon a problem where the predictable network device naming with systemd gave us some unpredictable results. The server has four onboard network ports (two 10GbE and two 1GbE) and an add-on 10GbE card with two additional ports.

Running lspci gives this output:

# lspci | grep Eth
01:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
01:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
08:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
08:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
42:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)
42:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)

If you’re not familiar with that output, it says:

Two 10GbE ports on PCI bus 1 (ports 0 and 1)
Two 1GbE ports on PCI bus 8 (ports 0 and 1)
Two 10GbE ports on PCI bus 42 (ports 0 and 1)

When the system boots up, the devices are named based on systemd-udevd’s criteria. Our devices looked like this after boot:

# ip addr | egrep ^[0-9]
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default
2: enp8s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
3: enp8s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
4: enp1s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
5: enp1s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
6: enp66s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
7: enp66s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000

Devices 2-5 make sense since they’re on PCI buses 1 and 8. However, our two-port NIC on PCI bus 42 has suddenly been named 66. We rebooted the server with the rd.udev.debug kernel command line to display debug messages from systemd-udevd during boot. That gave us this:

# journalctl | grep enp66s0f0
systemd-udevd[471]: renamed network interface eth0 to enp66s0f0
systemd-udevd[471]: NAME 'enp66s0f0' /usr/lib/udev/rules.d/80-net-setup-link.rules:13
systemd-udevd[471]: changing net interface name from 'eth0' to 'enp66s0f0'
systemd-udevd[471]: renamed netif to 'enp66s0f0'
systemd-udevd[471]: changed devpath to '/devices/pci0000:40/0000:40:02.0/0000:42:00.0/net/enp66s0f0'

So the system sees that the enp66s0f0 device is actually on PCI bus 42. What gives? A quick trip to #systemd on Freenode caused a facepalm:

mhayden | weird, udev shows it on pci bus 42 but yet names it 66
 jwl | 0x42 = 66

I didn’t expect to see hex. Sure enough, converting 42 in hex to decimal yields 66:

$ printf "%d\n" 0x42
66

That also helps to explain why the devices on buses 1 and 8 were unaffected. Converting 1 and 8 in hex to decimal gives 1 and 8. If you’re new to hex, this conversion table may help.

Photo Credit: mindfieldz via Compfight cc

Adventures in live booting Linux distributions

major@mhtx.net (Major Hayden) — Tue, 29 Jul 2014 13:05:54 +0000

We’re all familiar with live booting Linux distributions. Almost every Linux distribution under the sun has a method for making live CD’s, writing live USB sticks, or booting live images over the network. The primary use case for some distributions is on a live medium (like KNOPPIX).

However, I embarked on an adventure to look at live booting Linux for a different use case. Sure, many live environments are used for demonstrations or installations - temporary activities for a desktop or a laptop. My goal was to find a way to boot a large fleet of servers with live images. These would need to be long-running, stable, feature-rich, and highly configurable live environments.

Finding off the shelf solutions wasn’t easy. Finding cross-platform off the shelf solutions for live booting servers was even harder. I worked on a solution with a coworker to create a cross-platform live image builder that we hope to open source soon. (I’d do it sooner but the code is horrific.) ;)

Debian jessie (testing)

First off, we took a look at Debian’s Live Systems project. It consists of two main parts: something to build live environments, and something to help live environments boot well off the network. At the time of this writing, the live build process leaves a lot to be desired. There’s a peculiar tree of directories that are required to get started and the documentation isn’t terribly straightforward. Although there’s a bunch of documentation available, it’s difficult to follow and it seems to skip some critical details. (In all fairness, I’m an experienced Debian user but I haven’t gotten into the innards of Debian package/system development yet. My shortcomings there could be the cause of my problems.)

The second half of the Live Systems project consist of multiple packages that help with the initial boot and configuration of a live instance. These tools work extremely well. Version 4 (currently in alpha) has tools for doing all kinds of system preparation very early in the boot process and it’s compatible with SysVinit or systemd. The live images boot up with a simple SquashFS (mounted read only) and they use AUFS to add on a writeable filesystem that stays in RAM. Reads and writes to the RAM-backed filesystem are extremely quick and you don’t run into a brick wall when the filesystem fills up (more on that later with Fedora).

Ubuntu 14.04

Ubuntu uses casper which seems to precede Debian’s Live Systems project or it could be a fork (please correct me if I’m incorrect). Either way, it seemed a bit less mature than Debian’s project and left a lot to be desired.

Fedora and CentOS

Fedora 20 and CentOS 7 are very close in software versions and they use the same mechanisms to boot live images. They use dracut to create the initramfs and there are a set of dmsquash modules that handle the setup of the live image. The livenet module allows the live images to be pulled over the network during the early part of the boot process.

Building the live images is a little tricky. You’ll find good documentation and tools for standard live bootable CD’s and USB sticks, but booting a server isn’t as straightforward. Dracut expects to find a squashfs which contains a filesystem image. When the live image boots, that filesystem image is connected to a loopback device and mounted read-only. A snapshot is made via device mapper that gives you a small overlay for adding data to the live image.

This overlay comes with some caveats. Keeping tabs on how quickly the overlay is filling up can be tricky. Using tools like df is insufficient since device mapper snapshots are concerned with blocks. As you write 4k blocks in the overlay, you’ll begin to fill the snapshot, just as you would with an LVM snapshot. When the snapshot fills up and there are no blocks left, the filesystem in RAM becomes corrupt and unusable. There are some tricks to force it back online but I didn’t have much luck when I tried to recover. The only solution I could find was to hard reboot.

Arch

The ArchLinux live boot environments seem very similar to the ones I saw in Fedora and CentOS. All of them use dracut and systemd, so this makes sense. Arch once used a project called Larch to create live environments but it’s fallen out of support due to AUFS2 being removed (according to the wiki page).

Although I didn’t build a live environment with Arch, I booted one of their live ISO’s and found their live environment to be much like Fedora and CentOS. There was a device mapper snapshot available as an overlay and once it’s full, you’re in trouble.

OpenSUSE

The path to live booting an OpenSUSE image seems quite different. The live squashfs is mounted read only onto /read-only. An ext3 filesystem is created in RAM and is mounted on /read-write. From there, overlayfs is used to lay the writeable filesystem on top of the read-only squashfs. You can still fill up the overlay filesystem and cause some temporary problems, but you can back out those errant files and still have a useable live environment.

Here’s the problem: overlayfs was given the green light for consideration in the Linux kernel by Linus in 2013. It’s been proposed for several kernel releases and it didn’t make it into 3.16 (which will be released soon). OpenSUSE has wedged overlayfs into their kernel tree just as Debian and Ubuntu have wedged AUFS into theirs.

Wrap-up

Building highly customized live images isn’t easy and running them in production makes it more challenging. Once the upstream kernel has a stable, solid, stackable filesystem, it should be much easier to operate a live environment for extended periods. There has been a parade of stackable filesystems over the years (remember funion-fs?) but I’ve been told that overlayfs seems to be a solid contender. I’ll keep an eye out for those kernel patches to land upstream but I’m not going to hold my breath quite yet.