Inspecting OpenShift cgroups from inside the pod

My team at Red Hat builds a lot of kernels in OpenShift pods as part of our work with the Continuous Kernel Integration (CKI) project. We have lots of different pod sizes depending on the type of work we are doing and our GitLab runners spawn these pods based on the tags in our GitLab CI pipeline. Compiling with make When you compile a large software project, such as the Linux kernel, you can use multiple CPU cores to speed up the build....

2019-04-05 · 5 min · Major Hayden

Running Ansible in OpenShift with arbitrary UIDs

My work at Red Hat involves testing lots and lots of kernels from various sources and we use GitLab CE to manage many of our repositories and run our CI jobs. Those jobs run in thousands of OpenShift containers that we spawn every day. OpenShift has some handy security features that we like. First, each container is mounted read-only with some writable temporary space (and any volumes that you mount). Also, OpenShift uses arbitrarily assigned user IDs (UIDs) for each container....

2019-03-22 · 3 min · Major Hayden

Ensuring keepalived starts after the network is ready

After a recent OpenStack-Ansible (OSA) deployment on CentOS, I found that keepalived was not starting properly at boot time: Keepalived_vrrp[801]: Cant find interface br-mgmt for vrrp_instance internal !!! Keepalived_vrrp[801]: Truncating auth_pass to 8 characters Keepalived_vrrp[801]: VRRP is trying to assign ip address 172.29.236.11/32 to unknown br-mgmt interface !!! go out and fix your conf !!! Keepalived_vrrp[801]: Cant find interface br-mgmt for vrrp_instance external !!! Keepalived_vrrp[801]: Truncating auth_pass to 8 characters Keepalived_vrrp[801]: VRRP is trying to assign ip address 192....

2017-12-15 · 2 min · Major Hayden

Changes in RHEL 7 Security Technical Implementation Guide Version 1, Release 3

The latest release of the Red Hat Enterprise Linux Security Technical Implementation Guide (STIG) was published last week. This release is Version 1, Release 3, and it contains four main changes: V-77819 - Multifactor authentication is required for graphical logins V-77821 - Datagram Congestion Control Protocol (DCCP) kernel module must be disabled V-77823 - Single user mode must require user authentication V-77825 - Address space layout randomization (ASLR) must be enabled Deep dive Let’s break down this list to understand what each one means....

2017-11-02 · 3 min · Major Hayden

Import RPM repository GPG keys from other keyservers temporarily

I’ve been working through some patches to OpenStack-Ansible lately to optimize how we configure yum repositories in our deployments. During that work, I ran into some issues where pgp.mit.edu was returning 500 errors for some requests to retrieve GPG keys. Ansible was returning this error: curl: (22) The requested URL returned error: 502 Proxy Error error: http://pgp.mit.edu:11371/pks/lookup?op=get&search=0x61E8806C: import read failed(2) How does the rpm command know which keyserver to use? Let’s use the --showrc argument to show how it is configured:...

2017-09-20 · 2 min · Major Hayden

Apply the STIG to even more operating systems with ansible-hardening

Tons of improvements made their way into the ansible-hardening role in preparation for the OpenStack Pike release next month. The role has a new name, new documentation and extra tests. The role uses the Security Technical Implementation Guide (STIG) produced by the Defense Information Systems Agency (DISA) and applies the guidelines to Linux hosts using Ansible. Every control is configurable via simple Ansible variables and each control is thoroughly documented....

2017-07-21 · 2 min · Major Hayden

Old role, new name: ansible-hardening

The interest in the openstack-ansible-security role has taken off faster than I expected, and one piece of constant feedback I received was around the name of the role. Some users were unsure if they needed to use the role in an OpenStack cloud or if the OpenStack-Ansible project was required. The role works everywhere - OpenStack cloud or not. I started a mailing list thread on the topic and we eventually settled on a new name: ansible-hardening!...

2017-06-27 · 1 min · Major Hayden

Enable AppArmor on a Debian Jessie cloud image

I merged some initial Debian support into the openstack-ansible-security role and ran into an issue enabling AppArmor. The apparmor service failed to start and I found this output in the system journal: kernel: AppArmor: AppArmor disabled by boot time parameter Digging in That was unexpected. I was using the Debian jessie cloud image and it uses extlinux as the bootloader. The file didn’t reference AppArmor at all: # cat /boot/extlinux/extlinux.conf default linux timeout 1 label linux kernel boot/vmlinuz-3....

2017-05-24 · 3 min · Major Hayden

OpenStack Summit Boston 2017 Recap

The OpenStack Summit wrapped up today in Boston and it was a great week! There were plenty of informative breakouts and some interesting keynotes. Keynotes Beth Cohen shared some of the work that Verizon has done with software-defined WAN on customer-premises equipment (CPE). She showed a demo of how customers could easily provision virtual network hardware, such as firewalls or intrusion detection systems, without waiting for hardware or cabling changes. I’m less familiar with the world of telcos, so I found this really interesting....

2017-05-12 · 2 min · Major Hayden

OpenStack-Ansible networking on CentOS 7 with systemd-networkd

Although OpenStack-Ansible doesn’t fully support CentOS 7 yet, the support is almost ready. I have a four node Ocata cloud deployed on CentOS 7, but I decided to change things around a bit and use systemd-networkd instead of NetworkManager or the old rc scripts. This post will explain how to configure the network for an OpenStack-Ansible cloud on CentOS 7 with systemd-networkd. Each one of my OpenStack hosts has four network interfaces and each one has a specific task:...

2017-04-13 · 3 min · Major Hayden