Selinux on Major Hayden

What I learned while securing Ubuntu

major@mhtx.net (Major Hayden) — Wed, 14 Oct 2015 20:53:12 +0000

The blog posts have slowed down a bit lately because I’ve been heads down on a security project at work. I’m working with people in the OpenStack community to create a new Ansible role called openstack-ansible-security. The role aims to improve host security by using hardening standards to improve the configuration of various parts of the operating system.

This means applying security hardening to Ubuntu 14.04 systems since that’s the only host operating system supported by openstack-ansible at the moment. I have plenty of experience with securing Red Hat-based systems like Red Hat Enteprise Linux, CentOS and Fedora; but Ubuntu is new territory entirely. The rest of this post is full of lessons learned along the way.

Searching for hardening standards

Finding a complete hardening standard for Ubuntu 14.04 is challenging. The Center for Internet Security offers Ubuntu security benchmarks with two big caveats:

There are very few controls to apply (relative to what’s available for RHEL)
The terms of use are highly restrictive (no derivative works allowed)

With that idea off the table, I examined the other options that meet Requirement 2.2 of PCI-DSS 3.1 [PDF]. Anther choice was ISO 27002, but it’s not terribly specific or easy to automate with scripts. The same goes for NIST 800-53.

After plenty of searching, the decision was made to go forth with the Security Technical Implementation Guide (STIG) from the Defense Information Systems Agency (DISA) (part of the US Department of Defense). The STIGs aren’t licensed and they’re in the public domain. The only downside is that the closest STIG for use with Ubuntu 14.04 is the RHEL 6 STIG.

Using the RHEL 6 STIG meant that plenty of things will need to be translated for the different tools, configuration files, and package names that come with Ubuntu. It was frustrating to search all over for a hardening standard that applies well to Ubuntu and comes with decent auditing tools, but this was the best we could find.

Automatically starting daemons

The standard Ubuntu and Debian practice of automatically starting daemons has perplexed me before and it still continues to do so. Starting a daemon before I’ve had a chance to configure it makes little sense. The main argument is that the daemons come up with a highly secure configuration, so starting it automatically shouldn’t be a big deal. I’d prefer to install a package, have a look at the configuration, alter the configuration, and then start the daemon. Also, it had better not start after a reboot unless I explicitly ask it to do so.

There are plenty of examples where automatically starting a daemon with its default configuration is a bad idea. Take the postfix package as an example. If you install the package in non-interactive mode (as Ansible does by default), postfix will come online wth the following configuration option set:

inet_interfaces = all

Since Ubuntu doesn’t come with a firewall enabled by default, your postfix server is listening on all interfaces for mail immediately. The mynetworks configuration should prevent relaying, but any potential vulnerabilities in your postfix daemon are exposed to the network without your consent. I would prefer to configure postfix first before I ever allow it to run on my server.

Verifying packages

Say what you will about RPM packages and the rpm command, but the verification portions of the rpm command are quite helpful. Here’s an example of verifying the aide RPM in Fedora:

# rpm -Vv aide
......... c /etc/aide.conf
......... c /etc/logrotate.d/aide
......... /usr/sbin/aide
......... /usr/share/doc/aide
......... d /usr/share/doc/aide/AUTHORS
......... d /usr/share/doc/aide/COPYING
......... d /usr/share/doc/aide/ChangeLog
......... d /usr/share/doc/aide/NEWS
......... d /usr/share/doc/aide/README
......... d /usr/share/doc/aide/README.quickstart
......... /usr/share/doc/aide/contrib
......... d /usr/share/doc/aide/contrib/aide-attributes.sh
......... d /usr/share/doc/aide/contrib/bzip2.sh
......... d /usr/share/doc/aide/contrib/gpg2_check.sh
......... d /usr/share/doc/aide/contrib/gpg2_update.sh
......... d /usr/share/doc/aide/contrib/gpg_check.sh
......... d /usr/share/doc/aide/contrib/gpg_update.sh
......... d /usr/share/doc/aide/contrib/sshaide.sh
......... d /usr/share/doc/aide/manual.html
......... d /usr/share/man/man1/aide.1.gz
......... d /usr/share/man/man5/aide.conf.5.gz
......... /var/lib/aide
......... /var/log/aide

If the verification finds that nothing in the package has changed, it won’t print anything. I’ve added the -v here to ensure that everything is printed to the console. In the output, you can see that everything is checked. That includes configuration files, log directories, libraries, and documentation. If I change the content of the aide.conf by adding a comment, I see that change:

# echo "# Comment" >> /etc/aide.conf
# rpm -V aide
S.5....T. c /etc/aide.conf

The 5 denotes that the MD5 checksum on the file has changed since the package was installed. What happens if I change the owner, group, and mode of the aide.conf?

# chown major:major /etc/aide.conf
# rpm -V aide
S.5..UGT. c /etc/aide.conf

Now I have a UG there that denotes a user/group ownership change. Similar messages appear for changes to the permissions on files or directories. The restorecon command even lets you figure out when SELinux contexts have changed. If you set a file to have the wrong ownership or permission, one rpm command gets you back to normal:

# rpm --setperms --setugids aide

On the Ubuntu side, you can use the debsums package to help with some verification:

# debsums aide
/usr/bin/aide OK
/usr/share/doc/aide/NEWS.Debian.gz OK
/usr/share/doc/aide/changelog.Debian.gz OK
...
# debums aide-common
/usr/bin/aide-attributes OK
/usr/bin/aide.wrapper OK
/usr/sbin/aideinit OK
...

But wait — where are the configuration files? Where are the log and library directories? If you type these commands on an Ubuntu system, you’ll see that the configuration files and directories aren’t checked. In addition, there’s not a method for querying whether a particular file in a package has changed ownership or has had its mode changed. There’s also no option to restore the right permissions and ownership after an errant chown -R or chmod -R.

Managing AIDE

The AIDE package is critical for secure deployments since it helps administrators monitor for file integrity on a regular basis. However, Ubuntu ships with some interesting configuration files and wrappers for AIDE.

One of the unique configuration files is this one:

# cat /etc/aide/aide.conf.d/99_aide_root
/ Full

This causes AIDE to wander all over the system, indexing all types of files. It’s best to limit AIDE to a small number of directories whenever possible so that the AIDE runs complete quickly and the database file remains relatively small. Plenty of disk I/O can be used during AIDE runs, so it’s best to limit the scope.

Also, trying to initialize the database provides an unhelpful error:

# aide --init
Couldn't open file /var/lib/aide/please-dont-call-aide-without-parameters/aide.db.new for writing

That path doesn’t exist, and I’m confused because I did pass a parameter to aide. Long story short, you must use the aideinit command to initialize the aide database. That’s actually a bash script which then calls on aide.wrapper (another bash script) to actually run the aide binary for you. Better yet, aideinit is in /usr/sbin while aide.wrapper is in /usr/bin. This leads to plenty of confusion.

Linux Security Modules

It’s possible to run SELinux on Ubuntu, but the policies aren’t as well maintained as they are on other distributions. AppArmor is the recommended LSM on Ubuntu, but it doesn’t provide the granularity of SELinux. For example, SELinux confines almost every single process on a minimal Fedora system, but AppArmor confines almost nothing on a minimal Ubuntu-based system. AppArmor policies aren’t terribly restrictive and it’s possible to work around them due to their reliance on path names.

Fortunately, both LSM’s provide decent coverage with virtual machines and containers (using libvirt’s sVirt capability).

Summary

The upside is that there is plenty of room for security improvements, especially around usability, in Ubuntu. Ubuntu-centric hardening standards are difficult to find and challenging to apply. Every distribution has its quirks and differences, but it seems like securing Ubuntu comes with more unusual hoops to jump through relative to Red Hat-based distributions, OpenSUSE, and even Arch.

I plan to open some bugs for some of these smaller issues in the coming days. However, some of the larger philosophical issues (like automatically starting daemons) will be tougher to tackle.

systemd in Fedora 22: Failed to restart service: Access Denied

major@mhtx.net (Major Hayden) — Fri, 18 Sep 2015 19:43:35 +0000

If you’re running Fedora 22 and you’ve recently updated to systemd-219-24.fc22, you might see errors like these:

# systemctl restart postfix
Failed to restart postfix.service: Access denied

Your audit logs will have entries like these:

type=USER_AVC msg=audit(1442602150.292:763): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc: denied { start } for auid=n/a uid=0 gid=0 path="/run/systemd/system/session-4.scope" cmdline="/usr/lib/systemd/systemd-logind" scontext=system_u:system_r:systemd_logind_t:s0 tcontext=system_u:object_r:systemd_unit_file_t:s0 tclass=service exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
type=USER_AVC msg=audit(1442602150.437:768): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc: denied { start } for auid=n/a uid=0 gid=0 path="/usr/lib/systemd/system/user@.service" cmdline="/usr/lib/systemd/systemd-logind" scontext=system_u:system_r:systemd_logind_t:s0 tcontext=system_u:object_r:systemd_unit_file_t:s0 tclass=service exe="/usr/lib/systemd/systemd" sauid=0 hostname=? addr=? terminal=?'
type=USER_AVC msg=audit(1442602150.440:769): pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='avc: denied { start } for auid=n/a uid=0 gid=0 path="/run/systemd/system/session-4.scope" cmdline="/usr/lib/systemd/systemd-logind" scontext=system_u:system_r:systemd_logind_t:s0

There’s a very active bug under review to get it fixed. As a workaround, you can re-execute systemd with the following command:

systemctl daemon-reexec

That should allow you to stop, start, and restart services properly again. Also, you’ll be able to switch runlevels for reboots and shutdowns.

Keep an eye on the bug for more details as they develop. Kudos to Kevin Fenzi for the workaround!

Chronicles of SELinux: Dealing with web content in unusual directories

major@mhtx.net (Major Hayden) — Thu, 10 Sep 2015 13:40:35 +0000

I’ve decided to start a series of posts called “Chronicles of SELinux” where I hope to educate more users on how to handle SELinux denials with finesse rather than simply disabling it entirely. To kick things off, I’ll be talking about dealing with web content in the first post.

First steps

If you’d like to follow along, simply hop onto a system running Fedora 21 (or later), CentOS 7 or Red Hat Enterprise Linux 7. We need SELinux in enforcing mode on the host, so be sure to check the status with getenforce. Depending on what getenforce returns, you’ll need to make adjustments:

Enforcing: No adjustments needed - you’re all set!
Permissive: Run setenforce 1 and adjust SELinux configuration file (see below)
Disabled: Adjust the SELinux configuration file and reboot (see below)

To enable enforcing mode in the SELinux configuration file, edit /etc/selinux/config and ensure your SELINUX line has enforcing:

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=enforcing

If getenforce returned Disabled earlier, you will need to reboot to get SELinux working. Also be sure that the selinux-policy-targeted package is installed and run fixfiles onboot -B to relabel the system on reboot (thanks to immanetize for the comment).

Let’s install httpd and create a developer user:

# For Fedora
dnf -y install httpd
# For CentOS/RHEL
yum -y install httpd

useradd developer
systemctl enable httpd
systemctl start httpd

On to the guide!

Hosting content in an unique directory

On Red Hat-based systems, httpd expects to find its content in /var/www/html, but some system administrators prefer to have content stored elsewhere on the system. It could be on a SAN or other remote storage, but it could also just be in a different directory to make things easier for the business.

Let’s consider a situation where the web content is hosted from /web/. We can create the directory:

[root@fedora22 ~]# mkdir -v /web
mkdir: created directory '/web'

We can edit /etc/httpd/conf/httpd.conf and set our new DocumentRoot:

#
# DocumentRoot: The directory out of which you will serve your
# documents. By default, all requests are taken from this directory, but
# symbolic links and aliases may be used to point to other locations.
#
DocumentRoot "/web"

Let’s reload the httpd configuration:

systemctl reload httpd

And now we can add some amazing web content:

 /web/index.html

It’s time to test our web server:

# curl -i localhost/index.html
HTTP/1.1 403 Forbidden
Date: Thu, 10 Sep 2015 12:54:19 GMT
Server: Apache/2.4.16 (Fedora)
Content-Length: 219
Content-Type: text/html; charset=iso-8859-1

Oh, come on. What’s with this 403 error?

Investigating the 403

The first step for any situation like this is to review some logs. Let’s check the logs for httpd:

[Thu Sep 10 12:55:04.541789 2015] [core:error] [pid 16597] (13)Permission denied: [client ::1:49860] AH00035: access to /index.html denied (filesystem path '/web/index.html') because search permissions are missing on a component of the path

Search permissions are missing? What? Let’s check the permissions on our web directory:

# ls -al /web
total 12
drwxr-xr-x. 2 root root 4096 Sep 10 12:53 .
dr-xr-xr-x. 19 root root 4096 Sep 10 12:51 ..
-rw-r--r--. 1 root root 21 Sep 10 12:54 index.html

The httpd user has the ability to get into the directory (o+x is set on /web/) and the httpd user can read the file (o+r is set on /web/index.html). Let’s check the system journal just in case:

# journalctl -n 1 | tail
-- Logs begin at Thu 2015-09-10 12:31:37 UTC, end at Thu 2015-09-10 12:55:04 UTC. --
Sep 10 12:55:04 fedora22 audit[16597]: <audit-1400> avc: denied { getattr } for pid=16597 comm="httpd" path="/web/index.html" dev="xvda1" ino=524290 scontext=system_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:default_t:s0 tclass=file permissive=0

That’s quite a long log line. Let’s break it into pieces:

avc: denied { getattr } for pid=16597 comm="httpd": httpd tried to do something and was denied
path="/web/index.html" dev="xvda1" ino=524290: path to the file (index.html) involved in the denial
scontext=system_u:system_r:httpd_t:s0: the SELinux context of the httpd process
tcontext=unconfined_u:object_r:default_t:s0: the SELinux contect that is actually applied to our index.html
tclass=file: the denial came from accessing a file (index.html)
permissive=0: we’re in enforcing mode, not permissive mode

Long story short, when httpd tried to access our /web/index.html file, the httpd process was labeled with httpd_t, but the kernel found that the HTML file was labeled with default_t. The httpd process (labeled with httpd_t) isn’t allowed to read files that are labeled as default_t, so the access is denied.

Fixing it the right way

Since we know what SELinux expects for this file (from the log line in the journal), we can apply the right context and re-test. The chcon command has a handy argument that allows you to reference a file or directory, and apply the contexts from there. Since we know that /var/www/html has the right contexts already, we can use it as a reference:

# chcon -v -R --reference=/var/www/html /web
changing security context of '/web/index.html'
changing security context of '/web'

Now we see some different contexts on /web:

# ls -alZ /web/
total 12
drwxr-xr-x. 2 root root system_u:object_r:httpd_sys_content_t:s0 4096 Sep 10 13:19 .
dr-xr-xr-x. 19 root root system_u:object_r:root_t:s0 4096 Sep 10 13:19 ..
-rw-r--r--. 1 root root system_u:object_r:httpd_sys_content_t:s0 21 Sep 10 13:19 index.html

Let’s test again:

# curl -I localhost/index.html
HTTP/1.1 403 Forbidden
Date: Thu, 10 Sep 2015 13:21:22 GMT
Server: Apache/2.4.16 (Fedora)
Content-Type: text/html; charset=iso-8859-1

Darn! What’s in the httpd logs?

[Thu Sep 10 13:21:22.267719 2015] [authz_core:error] [pid 16593] [client ::1:49861] AH01630: client denied by server configuration: /web/index.html

Ah, we cleared the SELinux problem but now httpd is upset. Just below the DocumentRoot line that we edited earlier, look for two Directory blocks. Change /var/www/ and /var/www/html to /web in those blocks. Reload the httpd configuration and try once more:

# systemctl reload httpd
# curl -I localhost/index.html
HTTP/1.1 200 OK
Date: Thu, 10 Sep 2015 13:25:16 GMT
Server: Apache/2.4.16 (Fedora)
Last-Modified: Thu, 10 Sep 2015 13:19:47 GMT
ETag: "15-51f6474064d50"
Accept-Ranges: bytes
Content-Length: 21
Content-Type: text/html; charset=UTF-8

Success!

Long term fix

The chcon method is good for fixing one-off issues and for testing, but we need a good long term fix. SELinux has some file contexts already configured for certain directories, but not for our custom web directory. You can examine the defaults here:

# semanage fcontext -l | grep ^/var/www/html
/var/www/html(/.*)?/sites/default/files(/.*)? all files system_u:object_r:httpd_sys_rw_content_t:s0
/var/www/html(/.*)?/sites/default/settings\.php regular file system_u:object_r:httpd_sys_rw_content_t:s0
/var/www/html(/.*)?/uploads(/.*)? all files system_u:object_r:httpd_sys_rw_content_t:s0
/var/www/html(/.*)?/wp-content(/.*)? all files system_u:object_r:httpd_sys_rw_content_t:s0
/var/www/html/[^/]*/cgi-bin(/.*)? all files system_u:object_r:httpd_sys_script_exec_t:s0
/var/www/html/cgi/munin.* all files system_u:object_r:munin_script_exec_t:s0
/var/www/html/configuration\.php all files system_u:object_r:httpd_sys_rw_content_t:s0
/var/www/html/munin(/.*)? all files system_u:object_r:munin_content_t:s0
/var/www/html/munin/cgi(/.*)? all files system_u:object_r:munin_script_exec_t:s0
/var/www/html/owncloud/data(/.*)? all files system_u:object_r:httpd_sys_rw_content_t:s0

SELinux’s tools have a concept of equivalency. This allows you to say that one directory is equivalent to another one in the long term. We already used chcon to apply contexts with a reference to a directory with valid contexts, but this equivalency concept gives us a longer term fix. Here’s the command to use:

semanage fcontext --add --equal /var/www /web

If we break this down, we’re saying we want to add a new file context where /web is equal to /var/www. This means we want the same SELinux contexts applied in the same places and want them treated equally. After running the semanage command, let’s make an index2.html file to test:

 /web/index2.html
# curl -I localhost/index2.html
HTTP/1.1 200 OK
Date: Thu, 10 Sep 2015 13:35:24 GMT
Server: Apache/2.4.16 (Fedora)
Last-Modified: Thu, 10 Sep 2015 13:34:11 GMT
ETag: "15-51f64a78266c8"
Accept-Ranges: bytes
Content-Length: 21
Content-Type: text/html; charset=UTF-8

Great! We didn’t have to use chcon this time around because we configured /web as an equivalent directory to /var/www. Let’s double check the contexts:

# ls -alZ /web
total 16
drwxr-xr-x. 2 root root unconfined_u:object_r:httpd_sys_content_t:s0 4096 Sep 10 13:34 .
dr-xr-xr-x. 19 root root system_u:object_r:root_t:s0 4096 Sep 10 13:33 ..
-rw-r--r--. 1 root root unconfined_u:object_r:httpd_sys_content_t:s0 21 Sep 10 13:34 index2.html
-rw-r--r--. 1 root root unconfined_u:object_r:httpd_sys_content_t:s0 21 Sep 10 13:33 index.html

Perfect! We now have all of the security benefits of SELinux in a completely custom web directory.

AVC: denied dyntransition from sshd

major@mhtx.net (Major Hayden) — Thu, 03 Jul 2014 19:52:51 +0000

I’ve been working with some Fedora environments in chroots and I ran into a peculiar SELinux AVC denial a short while ago:

avc: denied { dyntransition } for pid=809 comm="sshd" scontext=system_u:system_r:kernel_t:s0 tcontext=system_u:system_r:sshd_net_t:s0 tclass=process

The ssh daemon is running on a non-standard port but I verified that the port is allowed with semanage port -l. The target context of sshd_net_t from the AVC seems sensible for the ssh daemon. I started to wonder if a context wasn’t applied correctly to the sshd excutable itself, so I checked within the chroot:

# ls -alZ /usr/sbin/sshd
-rwxr-xr-x. 1 root root system_u:object_r:sshd_exec_t:SystemLow 652816 May 15 03:56 /usr/sbin/sshd

That’s what it should be. I double-checked my running server (which booted a squashfs containing the chroot) and saw something wrong:

# ls -alZ /usr/sbin/sshd
-rwxr-xr-x. root root system_u:object_r:file_t:s0 /usr/sbin/sshd

How did file_t get there? It turns out that I was using rsync to drag data out of the chroot and I forgot to use the --xattrs argument with rsync.

Fixing broken DNS lookups in spamassassin

major@mhtx.net (Major Hayden) — Fri, 20 Jun 2014 13:20:56 +0000

I talked about the joys of running my own mail server last week only to find that my mail server was broken yesterday. Spamassassin stopped doing DNS lookups for RBL and SPF checks.

I had one of these moments:

My logs looked like this:

plugin: eval failed: available_nameservers: No DNS servers available!
plugin: eval failed: available_nameservers: No DNS servers available!
rules: failed to run NO_DNS_FOR_FROM RBL test, skipping:
 (available_nameservers: [...] No DNS servers available!)
 (available_nameservers: [...] No DNS servers available!

My /etc/resolv.conf was correct and had two valid DNS servers listed. Also, the permissions set on /etc/resolv.conf were reasonable (0644) and the SELinux context applied to the file was appropriate (net_conf_t). Everything else on the system was able to resolve DNS records properly. Even an strace on the spamd process showed it reading /etc/resolv.conf successfully!

It was Google time. I put some snippets of my error output into the search bar and found a spamassassin bug report. Mark Martinec found the root cause of the bug:

Net::DNS version 0.76 changed the field name holding a set of nameservers in a Net::DNS::Resolver object: it used to be ’nameservers’, but is now split into two fields: ’nameserver4’ and ’nameserver6'.

Mail/SpamAssassin/DnsResolver.pm relied on the internal field name of a Net::DNS::Resolver object to obtain a default list of recursive name servers, so the change in Net::DNS broke that.

The patch from the bug report worked just fine on my Fedora 20 mail server. Be sure to restart spamd after making the change.

There’s a Fedora bug report as well.

If anyone is interested, I plan to write up my email configuration on Fedora soon for other folks to use. I might even make some ansible playbooks for it. ;)

Fedora update: Fedora’s spamassassin package has been updated to 3.4.0-7 and it fixes two bugs. You’ll find it in the stable repositories in a few days.

Text missing in chrome on Linux

major@mhtx.net (Major Hayden) — Sun, 18 May 2014 04:33:14 +0000

I’m in the process of trying Fedora 20 on my retina MacBook and I ran into a peculiar issue with Chrome. Some sites would load up normally and I could read everything on the page. Other sites would load up and only some of the text would be displayed. Images were totally unaffected.

It wasn’t this way on the initial installation of Fedora but it cropped up somewhere along the way as I installed software. Changing the configuration within Chrome wasn’t an option - I couldn’t even see any text on the configuration pages!

The only commonality I could find is that all pages that specified their own web fonts (like the pages on this site) loaded up perfectly. Everything was visible. However, on sites that tend to use whatever font is available in the browser (sites that specify a font family), the text was missing. A good example was The Aviation Herald.

I remembered installing some Microsoft core fonts via Fedy and I added in some patched powerline fonts to work with tmux. A quick check of the SELinux troubleshooter alerted me to the problem: the new fonts had the wrong SELinux labels applied and Chrome wasn’t allowed to access them.

I decided to relabel the whole filesystem:

restorecon -Rv /

The restorecon output was line after line of fonts that I had installed earlier in the evening. Once it finished running, I started Chrome and it was working just as I had expected.

Launch secure LXC containers on Fedora 20 using SELinux and sVirt

major@mhtx.net (Major Hayden) — Tue, 22 Apr 2014 04:11:00 +0000

Getting started with LXC is a bit awkward and I’ve assembled this guide for anyone who wants to begin experimenting with LXC containers in Fedora 20. As an added benefit, you can follow almost every step shown here when creating LXC containers on Red Hat Enterprise Linux 7 Beta (which is based on Fedora 19).

You’ll need a physical machine or a VM running Fedora 20 to get started. (You could put a container in a container, but things get a little dicey with that setup. Let’s just avoid talking about nested containers for now. No, really, I shouldn’t have even brought it up. Sorry about that.)

Prep Work

Start by updating all packages to the latest versions available:

yum -y upgrade

Verify that SELinux is in enforcing mode by running getenforce. If you see Disabled or Permissive, get SELinux into enforcing mode with a quick configuration change:

sed -i 's/^SELINUX=.*/SELINUX=enforcing/' /etc/selinux/config

I recommend installing setroubleshoot-server to make it easier to find the root cause of AVC denials:

Reboot now. This will ensure that SELinux comes up in enforcing mode (verify that with getenforce after reboot) and it ensures that auditd starts up sedispatch (for setroubleshoot).

Install management libraries and utilities

Let’s grab libvirt along with LXC support and a basic NAT networking configuration.

yum -y install libvirt-daemon-lxc libvirt-daemon-config-network

Launch libvirtd via systemd and ensure that it always comes up on boot. This step will also adjust firewalld for your containers and ensure that dnsmasq is serving up IP addresses via DHCP on your default NAT network.

systemctl start libvirtd.service
systemctl enable libvirtd.service

Bootstrap our container

Installing packages into the container’s filesystem will take some time.

yum -y --installroot=/var/lib/libvirt/filesystems/fedora20 --releasever=20 --nogpg install systemd passwd yum fedora-release vim-minimal openssh-server procps-ng iproute net-tools dhclient

This step fills in the filesystem with the necessary packages to run a Fedora 20 container. We now need to tell libvirt about the container we’ve just created.

virt-install --connect lxc:// --name fedora20 --ram 512 --filesystem /var/lib/libvirt/filesystems/fedora20/,/

At this point, libvirt will know enough about the container to start it and you’ll be connected to the console of the container! We need to adjust some configuration files within the container to use it properly. Detach from the console with CTRL-].

Let’s stop the container so we can make some adjustments.

virsh -c lxc:// shutdown fedora20

Get the container ready for production

Hop into your container and set a root password.

chroot /var/lib/libvirt/filesystems/fedora20 /bin/passwd root

We will be logging in as root via the console occasionally and we need to allow that access.

echo "pts/0" >> /var/lib/libvirt/filesystems/fedora20/etc/securetty

Since we will be using our NAT network with our auto-configured dnsmasq server (thanks to libvirt), we can configure a simple DHCP setup for eth0:

cat < < EOF > /var/lib/libvirt/filesystems/fedora20/etc/sysconfig/network
NETWORKING=yes
EOF
cat < < EOF > /var/lib/libvirt/filesystems/fedora20/etc/sysconfig/network-scripts/ifcfg-eth0
BOOTPROTO=dhcp
ONBOOT=yes
DEVICE=eth0
EOF

Using ssh makes the container a lot easier to manage, so let’s ensure that it starts when the container boots. (You could do this via systemctl after logging in at the console, but I’m lazy.)

chroot /var/lib/libvirt/filesystems/fedora20/
ln -s /usr/lib/systemd/system/sshd.service /etc/systemd/system/multi-user.target.wants/
exit

Launch!

Cross your fingers and launch the container.

virsh -c lxc:// start --console fedora20

You’ll be attached to the console during boot but don’t worry, hold down CTRL-] to get back to your host prompt. Check the dnsmasq leases to find your container’s IP address and you can login as root over ssh.

cat /var/lib/libvirt/dnsmasq/default.leases

Security

After logging into your container via ssh, check the process labels within the container:

# ps aufxZ
LABEL USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 1 0.0 1.3 47444 3444 ? Ss 03:18 0:00 /sbin/init
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 18 0.0 2.0 43016 5368 ? Ss 03:18 0:00 /usr/lib/systemd/systemd-journald
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 38 0.4 7.8 223456 20680 ? Ssl 03:18 0:00 /usr/bin/python -Es /usr/sbin/firewalld -
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 40 0.0 0.7 26504 2084 ? Ss 03:18 0:00 /usr/sbin/smartd -n -q never
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 41 0.0 0.4 19268 1252 ? Ss 03:18 0:00 /usr/sbin/irqbalance --foreground
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 44 0.0 0.6 34696 1636 ? Ss 03:18 0:00 /usr/lib/systemd/systemd-logind
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 46 0.0 1.8 267500 4832 ? Ssl 03:18 0:00 /sbin/rsyslogd -n
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 dbus 47 0.0 0.6 26708 1680 ? Ss 03:18 0:00 /bin/dbus-daemon --system --address=syste
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 rpc 54 0.0 0.5 41992 1344 ? Ss 03:18 0:00 /sbin/rpcbind -w
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 55 0.0 0.3 25936 924 ? Ss 03:18 0:00 /usr/sbin/atd -f
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 56 0.0 0.5 22728 1488 ? Ss 03:18 0:00 /usr/sbin/crond -n
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 60 0.0 0.2 6412 784 pts/0 Ss+ 03:18 0:00 /sbin/agetty --noclear -s console 115200
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 74 0.0 3.2 339808 8456 ? Ssl 03:18 0:00 /usr/sbin/NetworkManager --no-daemon
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 394 0.0 5.9 102356 15708 ? S 03:18 0:00 \_ /sbin/dhclient -d -sf /usr/libexec/nm
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 polkitd 83 0.0 4.4 514792 11548 ? Ssl 03:18 0:00 /usr/lib/polkit-1/polkitd --no-debug
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 rpcuser 110 0.0 0.6 46564 1824 ? Ss 03:18 0:00 /sbin/rpc.statd
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 111 0.0 1.3 82980 3620 ? Ss 03:18 0:00 /usr/sbin/sshd -D
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 409 0.0 1.9 131576 5084 ? Ss 03:18 0:00 \_ sshd: root@pts/1
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 413 0.0 0.9 115872 2592 pts/1 Ss 03:18 0:00 \_ -bash
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 438 0.0 0.5 123352 1344 pts/1 R+ 03:19 0:00 \_ ps aufxZ
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 411 0.0 0.8 44376 2252 ? Ss 03:18 0:00 /usr/lib/systemd/systemd --user
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 412 0.0 0.5 66828 1328 ? S 03:18 0:00 \_ (sd-pam)
system_u:system_r:virtd_lxc_t:s0-s0:c0.c1023 root 436 0.0 0.4 21980 1144 ? Ss 03:19 0:00 /usr/lib/systemd/systemd-hostnamed

You’ll notice something interesting if you run getenforce now within the container — SELinux is disabled. Actually, it’s not really disabled. The processing of SELinux policy is done on the host. The container isn’t able to see what’s going on outside of its own files and processes. The libvirt documentation for LXC hints at the importance of this isolation:

A suitably configured UID/GID mapping is a pre-requisite to making containers secure, in the absence of sVirt confinement.

In the absence of the “user” namespace being used, containers cannot be considered secure against exploits of the host OS. The sVirt SELinux driver provides a way to secure containers even when the “user” namespace is not used. The cost is that writing a policy to allow execution of arbitrary OS is not practical. The SELinux sVirt policy is typically tailored to work with an simpler application confinement use case, as provided by the “libvirt-sandbox” project.

This leads to something really critical to understand:

Containers don’t contain

Dan Walsh has a great post that goes into the need for sVirt and the protections it can provide when you need to be insulated from potentially dangerous virtual machines or containers. If a user is root inside a container, they’re root on the host as well. (There’s an exception: UID namespaces. But let’s not talk about that now. Oh great, first it was nested containers and now I brought up UID namespaces. Sorry again.)

Dan’s talk about securing containers hasn’t popped up on the Red Hat Summit presentations page quite yet but here are some notes that I took and then highlighted:

Containers don’t contain. The kernel doesn’t know about containers. Containers simply use kernel subsystems to carve up namespaces for applications.
Containers on Linux aren’t complete. Don’t compare directly to Solaris zones yet.
Running containers without Mandatory Access Control (MAC) systems like SELinux or AppArmor opens the door for full system compromise via untrusted applications and users within containers.

Using MAC gives you one extra barrier to keep a malicious container from getting higher levels of access to the underlying host. There’s always a chance that a kernel exploit could bypass MAC but it certainly raises the level of difficulty for an attacker and allows server operators extra time to react to alerts.

Confine untrusted users (including your children) with SELinux

major@mhtx.net (Major Hayden) — Fri, 05 Jul 2013 18:50:43 +0000

The confined user support in SELinux is handy for ensuring that users aren’t able to do something that they shouldn’t. It seems more effective and easier to use than most of the other methods I’ve seen before. Thanks to Dan for reminding me about this during his SELinux in the Enterprise talk from this year’s Red Hat Summit.

There are five main SELinux user types (and a handy chart in the Fedora documentation):

guest_u: - no X windows, no sudo, and no networking
xguest_u: - same as guest_u, but X is allowed and connectivity is allowed to web ports only (handy for kiosks)
user_u: - same as xguest_u, but networking isn’t restricted
staff_u: - same as user_u, but sudo is allowed (su isn’t allowed)
unconfined_u: - full access (this is the default)

One interesting thing to note is that all users are allowed to execute binary applications within their home directories by default. This can be switch off via some booleans (which I’ll demonstrate in a moment).

Let’s kick off a demonstration to show the power of these restrictions. First off, let’s get a list of the default configuration:

# semanage login -l

Login Name SELinux User MLS/MCS Range Service

__default__ unconfined_u s0-s0:c0.c1023 *
root unconfined_u s0-s0:c0.c1023 *
system_u system_u s0-s0:c0.c1023 *

By default, all new users come with no restrictions (as shown by unconfined_u). I’ll create a new user called selinuxtest and set a password. If I ssh to the server as the selinuxtest user, I see that I’m unconfined:

$ id -Z
unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

That’s what we expected. Let’s apply the strongest restrictions to this user and apply guest_u:

# semanage login -a -s guest_u selinuxtest

I’ll start a new ssh session as selinuxtest and try out some commands that I’d normally expect to work on a Linux server:

$ ping google.com
ping: icmp open socket: Permission denied
$ curl google.com
curl: (7) Failed to connect to 74.125.225.129: Permission denied
$ sudo su -
sudo: unable to change to sudoers gid: Operation not permitted
$ ./hello
Hello world
$ file hello
hello: ELF 64-bit LSB executable, x86-64, version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.32, BuildID[sha1]=0x5ffb25a7171c3338d6c76147cccc666ddc752dde, not stripped

The networking and sudo restrictions applied as we expected. However, I was able to compile a small “Hello World” binary in C and run it. That could become a problem for some servers. Let’s adjust a boolean that will restrict this activity:

# getsebool -a | grep exec_content
auditadm_exec_content --> on
guest_exec_content --> on
secadm_exec_content --> on
staff_exec_content --> on
sysadm_exec_content --> on
user_exec_content --> on
xguest_exec_content --> on
# setsebool guest_exec_content off

Now I try running the binary again as my selinuxtest user:

$ ./hello
-bash: ./hello: Permission denied

I can’t execute binary content in my home directory or in /tmp any longer after adjusting the boolean. Let’s switch selinuxtest to xguest_u:

# semanage login -a -s xguest_u selinuxtest

And now I’ll re-test as the selinuxtest user:

$ curl -si google.com | head -1
HTTP/1.1 301 Moved Permanently
$ ping google.com
ping: icmp open socket: Permission denied

I have full web connectivity but I can’t do anything else on the network. Now for a switch to user_u:

# semanage login -a -s user_u selinuxtest

And testing user_u with selinuxtest reveals:

$ ping -c 1 google.com
PING google.com (74.125.225.134) 56(84) bytes of data.
64 bytes from ord08s09-in-f6.1e100.net (74.125.225.134): icmp_seq=1 ttl=57 time=29.3 ms

--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 29.332/29.332/29.332/0.000 ms
$ curl -si google.com | head -n1
HTTP/1.1 301 Moved Permanently
$ sudo su -
sudo: PERM_SUDOERS: setresuid(-1, 1, -1): Operation not permitted

Networking is wide open but I still don’t have sudo. Let’s try staff_u:

# semanage login -a -s staff_u selinuxtest

Testing staff_u with selinuxtest gives me the expected results:

$ sudo su -
[sudo] password for selinuxtest:

I didn’t add selinuxtest to sudoers, so this command would fail. However, I’m actually allowed to execute it now.

These restrictions could be very helpful when dealing with users that you don’t fully trust on your system. You could use these restrictions to add a kiosk user to a Linux machine and allow family members or coworkers to surf the web using your device. In addition, you could use the restrictions as an extra layer of protection on heavily shared servers to prevent users from consuming resources or generating malicious traffic.

Installing the Xen hypervisor on Fedora 19

major@mhtx.net (Major Hayden) — Mon, 03 Jun 2013 04:27:43 +0000

It’s been a little while since I last posted about installing Xen on Fedora, so I figured that Fedora 19’s beta release was as good a time as any to write a new post. To get started, you’ll need to get Fedora 19 installed on your favorite hardware (or virtual machine).

Install the Xen hypervisor and tools. Also, ensure that both of the necessary daemons are running on each boot:

yum -y install xen xen-hypervisor xen-libs xen-runtime
chkconfig xend on
chkconfig xendomains on

You’ll notice that I didn’t start the daemons quite yet. We will need the xen hypervisor running before they will be of any use.

Now, let’s configure GRUB2. I wrote a quick post about these steps last year. The Xen kernel entry should already be configured (by grubby), but it’s not the default. Fixing that is a quick process:

# grep ^menuentry /boot/grub2/grub.cfg | cut -d "'" -f2
Fedora, with Linux 3.9.4-300.fc19.x86_64
Fedora, with Linux 0-rescue-4ea51ecfff4f4e64a5ec903c495ee5b6
Fedora, with Xen hypervisor
# grub2-set-default 'Fedora, with Xen hypervisor'
# grub2-editenv list
saved_entry=Fedora, with Xen hypervisor

At this point, you’re ready to reboot. After the reboot, verify that Xen is running:

# xm dmesg | head
 __ __ _ _ ____ ____ ____ __ _ ___
 \ \/ /___ _ __ | || | |___ \ |___ \ | ___| / _| ___/ |/ _ \
 \ // _ \ '_ \ | || |_ __) | __) |_|___ \ | |_ / __| | (_) |
 / \ __/ | | | |__ _| / __/ _ / __/|__|__) || _| (__| |\__, |
 /_/\_\___|_| |_| |_|(_)_____(_)_____| |____(_)_| \___|_| /_/

(XEN) Xen version 4.2.2 (mockbuild@phx2.fedoraproject.org) (gcc (GCC) 4.8.0 20130412 (Red Hat 4.8.0-2)) Fri May 17 19:39:53 UTC 2013
(XEN) Latest ChangeSet: unavailable
(XEN) Bootloader: GRUB 2.00
(XEN) Command line: placeholder

If you’re adventurous on the command line, you’re done here. However, I enjoy using virt-manager for quick access to virtual machines and I also like all of the scripting and remote administration capabilities that libvirt delivers. Let’s get the tools and daemons installed and running:

yum -y install virt-manager dejavu* xorg-x11-xauth
yum -y install libvirt-daemon-driver-network libvirt-daemon-driver-storage libvirt-daemon-xen
chkconfig libvirtd on
service libvirtd start

You’re now ready to use virt-manager to manage your virtual machines. Simply ssh to your hypervisor with X forwarding enabled (ssh -X hypervisor.mydomain.com) and run virt-manager. You won’t have a virtual network or bridge to use for virtual machines quite yet. You have two options: NAT your VM’s or configure a network bridge. I prefer the bridge but you may require something different in your environment.

For the NAT option (the easiest for beginners):

yum -y install libvirt-daemon-config-network libvirt-daemon-config-nwfilter
service libvirtd restart

For the network-bridge option, you’ll need to adjust your network scripts to create a bridge and add your primary network interface to the bridge. That’s a bit outside the scope of this post, but the Fedora Wiki and HowtoForge (ignore the KVM parts of their guide).

You now have a working Xen installation on Fedora 19!

FOR THOSE WHO EMBRACE SECURITY:

If you run SELinux in Enforcing mode, there’s still a lingering issue where SELinux prevents python (running under xend) from talking to block devices (like logical volumes). I opened a bug about a similar problem before but I need to open another one for the block device issue. If you’re itching for a workaround, you can force SELinux into permissive mode for the xend_t context only:

yum -y install selinux-policy-devel
semanage permissive -a xend_t

That’s not the best option for now, but it’s certainly better than setenforce 0. ;)

Presentation: Demystifying SELinux

major@mhtx.net (Major Hayden) — Wed, 29 May 2013 01:01:09 +0000

While rolling through my RSS feeds, I found a great presentation by David Quigley titled “Demystifying SELinux”. He makes come good comparisons between discretionary/mandatory access controls and dives into what makes SELinux useful. Basic troubleshooting commands are covered within the presentation as well.

You can find the presentation over on Speaker Deck. I’ve also mirrored a PDF copy here on the site.

UPDATE: If you’re going to OSCON 2013 this year, it appears that David will be presenting this topic during the event.

Reprint: Stop Disabling SELinux!

major@mhtx.net (Major Hayden) — Fri, 19 Apr 2013 05:52:23 +0000

This article appeared in SC Magazine and I’ve posted it here as well. For those of you who were left wanting more from my previous SELinux post, this should help. If it doesn’t help, leave a comment. ;)

The push to cloud transforms the way we apply information security principles to systems and applications. Perimeters of the past, secured heavily with traditional network devices in the outermost ring, lose effectiveness day by day. Shifting the focus to “defense in depth” brings the perimeter down to the individual cloud instances running your application. Security-Enhanced Linux, or SELinux, forms an effective part of that perimeter.

SELinux operates in the realm of mandatory access control, or MAC. The design of MAC involves placing constraints on what a user (a subject) can do to a particular object (a target) on the system. In contrast, discretionary access control, or DAC, allows a user with certain access to use discretion to limit or allow access to certain files, directories, or devices. You can set any file system permissions that you want but SELinux can override them with ease at the operating system level.

Consider a typical server running a web application. An attacker compromises the web application and executes malicious code via the web server daemon itself. SELinux has default policies that prevent the daemon from initiating communication on the network. That limits the attacker’s options to attack other services or servers.

In addition, SELinux sets policies on which files and directories the web server can access, regardless of any file system permissions. This protection limits the attacker’s access to other sensitive parts of the file system even if the administrator set the files to be readable to the world.

This is where SELinux shines. Oddly enough, this is the point where many system administrators actually disable SELinux on their systems.

Troubleshooting these events, called AVC denials, without some helpful tools is challenging and frustrating. Each denial flows into to your audit log as a cryptic message. Most administrators will check the usual suspects, like firewall rules and file system permissions. As frustration builds, they disable SELinux and notice that their application begins working as expected. SELinux remains disabled and hundreds of helpful policies lie dormant solely because one policy caused a problem.

Disabling SELinux without investigation frustrated me to the point where I started a site at stopdisablingselinux.com. The site is a snarky response to Linux administrators who reach for the disable switch as soon as SELinux gets in their way.

All jokes aside, here are some helpful tips to use SELinux effectively:

Use the setroubleshoot helpers to understand denials

Working through denials is easy with the setroubleshoot-server package. When a denial occurs, you still receive a cryptic log message in your audit logs. However, you also receive a message via syslog that is very easy to read. Your server can email you these messages as well. The message contains guidance about adjusting SELinux booleans, setting contexts, or generating new SELinux policies to work around a really unusual problem. When I say guidance, I mean that the tools give you commands to copy and paste to adjust your policies, booleans and contexts.

Review SELinux booleans for quick adjustments

Although the myriad of SELinux user-space tools isn’t within the scope of this article, getsebool and togglesebool deserve a mention. Frequently adjusted policies are controlled by booleans that are toggled on and off with togglesebool. Start with getsebool –a for a full list of booleans and then use togglesebool to enable or disable the policy.

Quickly restore file or directory contexts

Shuffling files or directories around a server can cause SELinux denials due to contexts not matching their original values. This happens to me frequently if I move a configuration file from one system to another. Correcting the context problem involves one of two simple commands. The restorecon command applies the default contexts specific to the file or directory. If you have a file in the directory with the correct context, use chcon to fix the context on the wrong file by giving it the path to the file with the correct context.

Here are some additional links with helpful SELinux documentation:

Seriously, stop disabling SELinux

major@mhtx.net (Major Hayden) — Tue, 16 Apr 2013 04:40:10 +0000

After many discussions with fellow Linux users, I’ve come to realize that most seem to disable SELinux rather than understand why it’s denying access. In an effort to turn the tide, I’ve created a new site as a public service to SELinux cowards everywhere: stopdisablingselinux.com.

Here are some relatively useful SELinux posts from the blog:

Edit: The goal of the post was to poke some fun at system administrators who disable SELinux immediately without learning how it works or why they’re seeing certain operations being denied. Obviously, if your particular workload or demands don’t allow for the use of SELinux, then I’m going to be the last person to encourage you to use it. Many system administrators have found that it doesn’t provide a good ratio of work required to benefit gained, which I totally understand.

SELinux, Xen, and block devices in Fedora 17

major@mhtx.net (Major Hayden) — Tue, 10 Jul 2012 05:05:33 +0000

If you try to run Xen without libvirt on Fedora 17 with SELinux in enforcing mode, you’ll be butting heads with SELinux in no time. You’ll probably be staring at something like this:

# xm create -c fedora17
Using config file "/etc/xen/fedora17".
Error: Disk isn't accessible

If you have setroubleshoot and setroubleshoot-server installed, you should have a friendly message in /var/log/messages telling you the source of the problem:

setroubleshoot: SELinux is preventing /usr/bin/python2.7 from read access on the blk_file dm-1.
For complete SELinux messages. run sealert -l 4d890105-d9a4-4b3e-a674-ba7e952942dc

The Xen daemon (the python process mentioned in the SELinux denial) is running with a context type of xend_t but the block device I’m trying to use for the VM has fixed_disk_device_t:

# ps axZ | grep xend
system_u:system_r:xend_t:s0 953 ? SLl 0:40 /usr/bin/python /usr/sbin/xend
# ls -alZ /dev/dm-1
brw-rw----. root disk system_u:object_r:fixed_disk_device_t:s0 /dev/dm-1

SELinux isn’t going to allow this to work. However, even if we fix this, SELinux will balk about three additional issues and we’ll need to adjust the contexts on every new fixed block device we make. To get over the hump, change the context type on your block device to xen_image_t and re-run the xm create:

# chcon -t xen_image_t /dev/dm-1
# ls -alZ /dev/dm-1
brw-rw----. root disk system_u:object_r:xen_image_t:s0 /dev/dm-1
# xm create -c fedora17
Using config file "/etc/xen/fedora17".
Error: out of pty devices

You’ll find three new denials in /var/log/messages:

setroubleshoot: SELinux is preventing /usr/bin/python2.7 from read access on the file group.
For complete SELinux messages. run sealert -l b1392df4-dda4-4b82-914c-1e20c62fc898
setroubleshoot: SELinux is preventing /usr/bin/python2.7 from setattr access on the chr_file 1.
For complete SELinux messages. run sealert -l 3e09edc3-aeb7-49f5-96e1-d8148afda48f
setroubleshoot: SELinux is preventing /usr/bin/python2.7 from execute access on the file pt_chown.
For complete SELinux messages. run sealert -l 86395f09-5f33-4f66-8d02-519b61e54139

As much as it pains me to suggest it, you can create a custom module to allow all four of these operations by xend:

# grep xend /var/log/audit/audit.log | audit2allow -M custom_xen
WARNING: Policy would be downgraded from version 27 to 26.
******************** IMPORTANT ***********************
To make this policy package active, execute:

semodule -i custom_xen.pp

# semodule -i custom_xen.pp

You should now be able to start your VM without any complaints from SELinux. I’ll reiterate that this isn’t ideal, but it’s the best balance of security and convenience that I’ve found so far.

Getting started with SELinux

major@mhtx.net (Major Hayden) — Thu, 26 Jan 2012 04:28:41 +0000

I used to be one of those folks who would install Fedora, CentOS, Scientific Linux, or Red Hat and disable SELinux during the installation. It always seemed like SELinux would get in my way and keep me from getting work done.

Later on, I found that one of my servers (which I’d previously secured quite thoroughly) had some rogue processes running that were spawned through httpd. Had I actually been using SELinux in enforcing mode, those processes would have probably never even started.

If you’re trying to get started with SELinux but you’re not sure how to do it without completely disrupting your server’s workflow, these tips should help:

Get some good reporting and monitoring

Two of the most handy SELinux tools are setroubleshoot and setroubleshoot-server. If you’re running a server without X, you can use my guide for configuring setroubleshoot-server. You will receive email alerts within seconds of an AVC denial and the emails should contain tips on how to resolve the denial if the original action should be allowed. If the AVC denial caught something you didn’t expect, you’ll know about the potential security breach almost immediately.

Start out with SELinux in permissive mode

If you’re overly concerned about SELinux getting in your way, or if you’re enabling SELinux on a server that has been running without SELinux since it was installed, start out with SELinux in permissive mode. To make the change effective immediately, just run:

# setenforce 0
# getenforce
Permissive

Edit /etc/sysconfig/selinux to make it persistent across reboots:

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=permissive

Adjust booleans before adding your own custom modules

There are a lot of booleans you can toggle to get the functionality you need without adding your own custom SELinux modules with audit2allow. If you wanted to see all of the applicable booleans for httpd, just use getsebool:

# getsebool -a | grep httpd
httpd_builtin_scripting --> on
httpd_can_check_spam --> off
httpd_can_network_connect --> on
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_can_network_memcache --> off
httpd_can_network_relay --> on
httpd_can_sendmail --> on
... and so on ...

Toggling booleans is easy with togglesebool:

# togglesebool httpd_can_network_memcache
httpd_can_network_memcache: active

Now httpd can talk to memcache. You can also use setsebool if you want to be specific about your setting (this is good for scripts):

# setsebool httpd_can_network_memcache on

Tracking your history of AVC denials

All of your AVC denals are logged by auditd in /var/log/audit/audit.log but it’s not the easiest file to read and parse. That’s where aureport comes in:

# aureport --avc | tail -n 5
45. 01/24/2012 04:23:29 postdrop unconfined_u:system_r:httpd_t:s0 4 fifo_file getattr system_u:object_r:postfix_public_t:s0 denied 1061
46. 01/24/2012 04:23:29 postdrop unconfined_u:system_r:httpd_t:s0 2 fifo_file write system_u:object_r:postfix_public_t:s0 denied 1062
47. 01/24/2012 04:23:29 postdrop unconfined_u:system_r:httpd_t:s0 2 fifo_file open system_u:object_r:postfix_public_t:s0 denied 1062
48. 01/24/2012 14:01:58 sendmail unconfined_u:system_r:httpd_t:s0 160 process setrlimit unconfined_u:system_r:httpd_t:s0 denied 1123
49. 01/24/2012 14:01:58 postdrop unconfined_u:system_r:httpd_t:s0 4 dir search system_u:object_r:postfix_public_t:s0 denied 1124

Summary

There’s no need to be scared of or be annoyed by SELinux in your server environment. While it takes some getting used to (and what new software doesn’t?), you’ll have an extra layer of security and access restrictions which should let you sleep a little better at night.

SELinux and .forward files

major@mhtx.net (Major Hayden) — Mon, 02 Jan 2012 22:44:43 +0000

If you want to forward e-mail from root to another user, you can usually place a .forward file in root’s home directory and your mail server will take care of the rest:

 /root/.forward

With SELinux, you’ll end up getting an AVC denial each time your mail server tries to read the contents of the .forward file:

type=AVC msg=audit(1325543823.787:7416): avc: denied { open } for pid=9850
 comm="local" name=".forward" dev=md0 ino=17694734
 scontext=system_u:system_r:postfix_local_t:s0
 tcontext=unconfined_u:object_r:admin_home_t:s0 tclass=file

The reason is that your .forward file doesn’t have the right SELinux contexts. You can set the correct contest quickly with restorecon:

# ls -Z /root/.forward
-rw-r--r--. root root unconfined_u:object_r:admin_home_t:s0 /root/.forward
# restorecon -v /root/.forward
restorecon reset /root/.forward context unconfined_u:object_r:admin_home_t:s0->system_u:object_r:mail_forward_t:s0
# ls -Z /root/.forward
-rw-r--r--. root root system_u:object_r:mail_home_t:s0 /root/.forward

Try to send another e-mail to root and you should see the mail server forward the e-mail properly without any additional AVC denials.

Receive e-mail reports for SELinux AVC denials

major@mhtx.net (Major Hayden) — Fri, 16 Sep 2011 04:17:04 +0000

SELinux isn’t a technology that’s easy to tackle for newcomers. However, there’s been a lot of work to smooth out the rough edges while still keeping a tight grip on what applications and users are allowed to do on a Linux system. One of the biggest efforts has been around setroubleshoot.

The purpose behind setroubleshoot is to let users know when access has been denied, help them resolve it if necessary, and to reduce overall frustration while working through tight security restrictions in the default SELinux policies. The GUI frontend for setroubleshoot is great for users who run Linux desktops or those who run servers with a display attached. Don’t worry, you can configure setroubleshoot on remote servers to send alerts elsewhere when a GUI alert isn’t an option.

Install a few packages to get started:

yum install setroubleshoot{-server,-plugins,-doc}

Open /etc/setroubleshoot/setroubleshoot.conf in your favorite text editor and adjust the [email] section to fit your server:

recipients_filepath = /var/lib/setroubleshoot/email_alert_recipients
smtp_port = 25
smtp_host = localhost
from_address = selinux@myserver.com
subject = [MyServer] SELinux AVC Alert

You could probably see it coming, but you need to put the e-mail addresses for your recipients into /var/lib/setroubleshoot/email_alert_recipients:

echo "selinux@mycompany.com" >> /var/lib/setroubleshoot/email_alert_recipients

You’ll notice that setroubleshoot doesn’t have an init script and it doesn’t exist in systemd in Fedora 15. It runs through the dbus-daemon and a quick bounce of the messagebus via its init script brings in the necessary components to run setroubleshoot:

service messagebus restart

A really easy (and safe) test is to ask sshd to bind to a non-standard port. Simply define an additional port on in your /etc/ssh/sshd_config like this:

Port 22
Port 222

When you restart sshd, it will bind to port 22 with success, but it won’t be allowed to bind to port 222 (since that’s blocked by SELinux as a non-standard port for the ssh_port_t port type). DON’T WORRY! Your sshd server will still be listening on port 22. If you wait a moment, you’ll get an e-mail (perhaps two) that not only notify you of the denial, but they make suggestions for how to fix it:

SELinux is preventing /usr/sbin/sshd from name_bind access on the tcp_socket port 222.

***** Plugin bind_ports (99.5 confidence) suggests *************************

If you want to allow /usr/sbin/sshd to bind to network port 222
Then you need to modify the port type.
Do
# semanage port -a -t PORT_TYPE -p tcp 222
 where PORT_TYPE is one of the following: ...

For this particular example, the quick fix would be to run:

semanage port -a -t ssh_port_t -p tcp 222

Much of this post’s information was gathered from the detailed documentation on Fedora’s setroubleshoot User’s FAQ as well as Dan Walsh’s setroubleshoot blog post.

Getting apache, PHP, and memcached working with SELinux

major@mhtx.net (Major Hayden) — Thu, 08 Sep 2011 03:55:00 +0000

I’m using SELinux more often now on my Fedora 15 installations and I came up against a peculiar issue today on a new server. My PHP installation is configured to store its sessions in memcached and I brought over some working configurations from another server. However, each time I accessed a page which tried to initiate a session, the page load would hang for about a minute and I’d find this in my apache error logs:

[Thu Sep 08 03:23:40 2011] [error] [client 11.22.33.44] PHP Warning:
Unknown: Failed to write session data (memcached). Please verify that
the current setting of session.save_path is correct (127.0.0.1:11211)
in Unknown on line 0

I ran through my usual list of checks:

netstat showed memcached bound to the correct ports/interfaces
memcached was running and I could reach it via telnet
memcached-tool could connect and pull stats from memcached
double-checked my php.ini
tested memcached connectivity via a PHP and ruby script - they worked

Even after all that, I still couldn’t figure out what was wrong. I ran strace on memcached while I ran a curl against the page which creates a session and I found something significant - memcached wasn’t seeing any connections whatsoever at that time. A quick check of the lo interface with tcpdump showed the same result. Just before I threw a chair, I remembered one thing:

SELinux.

A quick check for AVC denials showed the problem:

# aureport --avc | tail -n 1
4021. 09/08/2011 03:23:38 httpd system_u:system_r:httpd_t:s0 42 tcp_socket name_connect system_u:object_r:memcache_port_t:s0 denied 31536

I’m far from being a guru on SELinux, so I leaned on audit2allow for help:

# grep memcache /var/log/audit/audit.log | audit2allow

#============= httpd_t ==============
#!!!! This avc can be allowed using one of the these booleans:
# httpd_can_network_relay, httpd_can_network_memcache, httpd_can_network_connect

allow httpd_t memcache_port_t:tcp_socket name_connect;

The boolean we’re looking for is httpd_can_network_memcache. Flipping the boolean can be done in a snap:

# setsebool -P httpd_can_network_memcache 1
# getsebool httpd_can_network_memcache
httpd_can_network_memcache --> on

After adjusting the boolean, apache was able to make connections to memcached without a hitch. My page which created sessions loaded quickly and I could see data being stored in memcached. If you want to check the status of all of the apache-related SELinux booleans, just use getsebool:

# getsebool -a | grep httpd | grep off$
allow_httpd_anon_write --> off
allow_httpd_mod_auth_ntlm_winbind --> off
allow_httpd_mod_auth_pam --> off
allow_httpd_sys_script_anon_write --> off
httpd_can_check_spam --> off
httpd_can_network_connect_cobbler --> off
httpd_can_network_connect_db --> off
httpd_can_network_relay --> off
httpd_can_sendmail --> off
httpd_dbus_avahi --> off
httpd_enable_ftp_server --> off
httpd_enable_homedirs --> off
httpd_execmem --> off
httpd_read_user_content --> off
httpd_setrlimit --> off
httpd_ssi_exec --> off
httpd_tmp_exec --> off
httpd_unified --> off
httpd_use_cifs --> off
httpd_use_gpg --> off
httpd_use_nfs --> off

If you’re interested in SELinux, a good way to get your feet wet is to head over to the CentOS Wiki and review their SELinux Howtos