major.io

Words of wisdom from a Linux engineer focused on information security

major.io

Words of wisdom from a systems engineer

Creative Commons License

Old role, new name: ansible-hardening

By Leave a Comment

The interest in the openstack-ansible-security role has taken off faster than I expected, and one piece of constant feedback I received was around the name of the role. Some users were unsure if they needed to use the role in an OpenStack cloud or if the OpenStack-Ansible project was required.

ansible-hardening logoThe role works everywhere — OpenStack cloud or not. I started a mailing list thread on the topic and we eventually settled on a new name: ansible-hardening! The updated documentation is already available.

The old openstack-ansible-security role is being retired and it will not receive any additional updates. Moving to the new role is easy:

  1. Install ansible-hardening with ansible-galaxy (or git clone)
  2. Change your playbooks to use the ansible-hardening role

There’s no need to change any variable names or tags — they are all kept the same in the new role.

As always, if you have questions or comments about the role, drop by #openstack-ansible on Freenode IRC or open a bug in Launchpad.

Enable AppArmor on a Debian Jessie cloud image

By 1 Comment

Armor from the middle ages makes sense when you're talking about AppArmor, right?I merged some initial Debian support into the openstack-ansible-security role and ran into an issue enabling AppArmor. The apparmor service failed to start and I found this output in the system journal:

Digging in

That was unexpected. I was using the Debian jessie cloud image and it uses extlinux as the bootloader. The file didn’t reference AppArmor at all:

I learned that AppArmor is disabled by default in Debian unless you explicitly enable it. In contrast, SELinux is enabled unless you turn it off. To make matters worse, Debian’s cloud image doesn’t have any facilities or scripts to automatically update the extlinux configuration file when new kernels are installed.

Making a repeatable fix

My two goals here were to:

  1. Ensure AppArmor is enabled on the next boot
  2. Ensure that AppArmor remains enabled when new kernels are installed

The first step is to install grub2:

During the installation, a package configuration window will appear that asks about where grub should be installed. I selected /dev/vda from the list and waited for apt to finish the package installation.

The next step is to edit /etc/default/grub and add in the AppArmor configuration. Adjust the GRUB_CMDLINE_LINUX_DEFAULT line to look like the one below:

Ensure that the required AppArmor packages are installed:

Enable the AppArmor service upon reboot:

Run update-grub and reboot. After the reboot, run apparmor_status and you should see lots of AppArmor profiles loaded:

Final thoughts

I’m still unsure about why AppArmor is disabled by default. There aren’t that many profiles shipped by default (38 on my freshly installed jessie system versus 417 SELinux policies in Fedora 25) and many of them affect services that wouldn’t cause significant disruptions on the system.

There is a discussion that ended last year around how to automate the AppArmor enablement process when the AppArmor packages are installed. This would be a great first step to make the process easier, but it would probably make more sense to take the step of enabling it by default.

Photo credit: Max Pixel

Fixing OpenStack noVNC consoles that ignore keyboard input

By Leave a Comment

Televideo consoleI opened up a noVNC console to a virtual machine today in my OpenStack cloud but found that the console wouldn’t take keyboard input. The Send Ctrl-Alt-Del button in the top right of the window worked just fine, but I couldn’t type anywhere in the console. This happened on an Ocata OpenStack cloud deployed with OpenStack-Ansible on CentOS 7.

Test the network path

The network path to the console is a little deep for this deployment, but here’s a quick explanation:

  • My laptop connects to HAProxy
  • HAProxy sends the traffic to the nova-novncproxy service
  • nova-novncproxy connects to the correct VNC port on the right hypervisor

If all of that works, I get a working console! I knew the network path was set up correctly because I could see the console in my browser.

My next troubleshooting step was to dump network traffic with tcpdump on the hypervisor itself. I dumped the traffic on port 5900 (which was the VNC port for this particular instance) and watched the output. Whenever I wiggled the mouse over the noVNC console in my browser, I saw a flurry of network traffic. The same thing happened if I punched lots of keys on the keyboard. At this point, it was clear that the keyboard input was making it to the hypervisor, but it wasn’t being handled correctly.

Test the console

Next, I opened up virt-manager, connected to the hypervisor, and opened a connection to the instance. The keyboard input worked fine there. I opened up remmina and connected via plain old VNC. The keyboard input worked fine there, too!

Investigate in the virtual machine

The system journal in the virtual machine had some interesting output:

It seems like my keyboard input was being lost in translation — literally. I have a US layout keyboard (Thinkpad X1 Carbon) and the virtual machine was configured with the en-us keymap:

A thorough Googling session revealed that it is not recommended to set a keymap for virtual machines in libvirt in most situations. I set the nova_console_keymap variable in /etc/openstack_deploy/user_variables.yml to an empty string:

I redeployed the nova service using the OpenStack-Ansible playbooks:

Once that was done, I powered off the virtual machine and powered it back on. (This is needed to ensure that the libvirt changes go into effect for the virtual machine.)

Great success! The keyboard was working in the noVNC console once again!

Photo credit: Wikipedia

OpenStack Summit Boston 2017 Recap

By Leave a Comment

Bunker Hill Monument Boston OpenStack SummitThe OpenStack Summit wrapped up today in Boston and it was a great week! There were plenty of informative breakouts and some interesting keynotes.

Keynotes

Beth Cohen shared some of the work that Verizon has done with software-defined WAN on customer-premises equipment (CPE). She showed a demo of how customers could easily provision virtual network hardware, such as firewalls or intrusion detection systems, without waiting for hardware or cabling changes. I’m less familiar with the world of telcos, so I found this really interesting.

Daniela Rus gave an amazing keynote about the democratization of robotics. She showed videos of tiny robots doing some amazing things, including robots which could be swallowed. Those robots could help children who swallow dangerous things (like batteries) without painful surgery.

The big surprise on the second day was the Q&A with Edward Snowden. At first, I was skeptical about it being a publicity stunt, but it turned out to be a really good conversation about the value of open source.

My favorite keynote was from Patrick Weeks of GE. He talked about their IT transformation goals and how they selected OpenStack to solve them. They chose a solution from Rackspace and their engineers love it!

Breakouts

Here are some links to my favorite breakouts:

OpenStack-Ansible

Although I couldn’t make it to all of the OpenStack-Ansible sessions, we had a great turnout for the ones I attended! Every seat was taken during the developer onboarding session and we had some helpful comments from new contributors.

Andy McCrae leads the OpenStack-Ansible onboarding session

Andy McCrae leads the OpenStack-Ansible onboarding session

My talks

The week was a long one for me! I shared two full-length talks, helped with a lightning talk, and joined a panel. Here are some quick links to the videos and slides:

  • Grow Your Community: Inspire an Impostor
  • Securing OpenStack Cloud and Beyond with Ansible
  • The Open Open Open Open Cloud
  • OpenStack Security Team Update

Photo credit: Luciot

OpenStack-Ansible networking on CentOS 7 with systemd-networkd

By 1 Comment

Ethernet switchAlthough OpenStack-Ansible doesn’t fully support CentOS 7 yet, the support is almost ready. I have a four node Ocata cloud deployed on CentOS 7, but I decided to change things around a bit and use systemd-networkd instead of NetworkManager or the old rc scripts.

This post will explain how to configure the network for an OpenStack-Ansible cloud on CentOS 7 with systemd-networkd.

Each one of my OpenStack hosts has four network interfaces and each one has a specific task:

  • enp2s0 – regular network interface, carries inter-host LAN traffic
  • enp3s0 – carries br-mgmt bridge for LXC container communication
  • enp4s0 – carries br-vlan bridge for VM public network connectivity
  • enp5s0 – carries br-vxlan bridge for VM private network connectivity

Adjusting services

First off, we need to get systemd-networkd and systemd-resolved ready to take over networking:

LAN interface

My enp2s0 network interface carries traffic between hosts and handles regular internal LAN traffic.

This one is quite simple, but the rest get a little more complicated.

Management bridge

The management bridge (br-mgmt) carries traffic between LXC containers. We start by creating the bridge device itself:

Now we configure the network on the bridge (I use OpenStack-Ansible’s defaults here):

I run the management network on VLAN 10, so I need a network device and network configuration for the VLAN as well. This step adds the br-mgmt bridge to the VLAN 10 interface:

Finally, we add the VLAN 10 interface to enp3s0 to tie it all together:

Public instance connectivity

My router offers up a few different VLANs for OpenStack instances to use for their public networks. We start by creating a br-vlan network device and its configuration:

We can add this bridge onto the enp4s0 physical interface:

VXLAN private instance connectivity

This step is similar to the previous one. We start by defining our br-vxlan bridge:

My VXLAN traffic runs over VLAN 11, so we need to define that VLAN interface:

We can hook this VLAN interface into the enp5s0 interface now:

Checking our work

The cleanest way to apply all of these configurations is to reboot. The Adjusting services step from the beginning of this post will ensure that systemd-networkd and systemd-resolved come up after a reboot.

Run networkctl to get a current status of your network interfaces:

You should have configured in the SETUP column for all of the interfaces you created. Some interfaces will show as degraded because they are missing an IP address (which is intentional for most of these interfaces).