major.io

Words of wisdom from a Linux engineer focused on information security

major.io

Words of wisdom from a systems engineer

Creative Commons License

Fixing OpenStack noVNC consoles that ignore keyboard input

By Leave a Comment

Televideo consoleI opened up a noVNC console to a virtual machine today in my OpenStack cloud but found that the console wouldn’t take keyboard input. The Send Ctrl-Alt-Del button in the top right of the window worked just fine, but I couldn’t type anywhere in the console. This happened on an Ocata OpenStack cloud deployed with OpenStack-Ansible on CentOS 7.

Test the network path

The network path to the console is a little deep for this deployment, but here’s a quick explanation:

  • My laptop connects to HAProxy
  • HAProxy sends the traffic to the nova-novncproxy service
  • nova-novncproxy connects to the correct VNC port on the right hypervisor

If all of that works, I get a working console! I knew the network path was set up correctly because I could see the console in my browser.

My next troubleshooting step was to dump network traffic with tcpdump on the hypervisor itself. I dumped the traffic on port 5900 (which was the VNC port for this particular instance) and watched the output. Whenever I wiggled the mouse over the noVNC console in my browser, I saw a flurry of network traffic. The same thing happened if I punched lots of keys on the keyboard. At this point, it was clear that the keyboard input was making it to the hypervisor, but it wasn’t being handled correctly.

Test the console

Next, I opened up virt-manager, connected to the hypervisor, and opened a connection to the instance. The keyboard input worked fine there. I opened up remmina and connected via plain old VNC. The keyboard input worked fine there, too!

Investigate in the virtual machine

The system journal in the virtual machine had some interesting output:

It seems like my keyboard input was being lost in translation — literally. I have a US layout keyboard (Thinkpad X1 Carbon) and the virtual machine was configured with the en-us keymap:

A thorough Googling session revealed that it is not recommended to set a keymap for virtual machines in libvirt in most situations. I set the nova_console_keymap variable in /etc/openstack_deploy/user_variables.yml to an empty string:

I redeployed the nova service using the OpenStack-Ansible playbooks:

Once that was done, I powered off the virtual machine and powered it back on. (This is needed to ensure that the libvirt changes go into effect for the virtual machine.)

Great success! The keyboard was working in the noVNC console once again!

Photo credit: Wikipedia

OpenStack Summit Boston 2017 Recap

By Leave a Comment

Bunker Hill Monument Boston OpenStack SummitThe OpenStack Summit wrapped up today in Boston and it was a great week! There were plenty of informative breakouts and some interesting keynotes.

Keynotes

Beth Cohen shared some of the work that Verizon has done with software-defined WAN on customer-premises equipment (CPE). She showed a demo of how customers could easily provision virtual network hardware, such as firewalls or intrusion detection systems, without waiting for hardware or cabling changes. I’m less familiar with the world of telcos, so I found this really interesting.

Daniela Rus gave an amazing keynote about the democratization of robotics. She showed videos of tiny robots doing some amazing things, including robots which could be swallowed. Those robots could help children who swallow dangerous things (like batteries) without painful surgery.

The big surprise on the second day was the Q&A with Edward Snowden. At first, I was skeptical about it being a publicity stunt, but it turned out to be a really good conversation about the value of open source.

My favorite keynote was from Patrick Weeks of GE. He talked about their IT transformation goals and how they selected OpenStack to solve them. They chose a solution from Rackspace and their engineers love it!

Breakouts

Here are some links to my favorite breakouts:

OpenStack-Ansible

Although I couldn’t make it to all of the OpenStack-Ansible sessions, we had a great turnout for the ones I attended! Every seat was taken during the developer onboarding session and we had some helpful comments from new contributors.

Andy McCrae leads the OpenStack-Ansible onboarding session

Andy McCrae leads the OpenStack-Ansible onboarding session

My talks

The week was a long one for me! I shared two full-length talks, helped with a lightning talk, and joined a panel. Here are some quick links to the videos and slides:

  • Grow Your Community: Inspire an Impostor
  • Securing OpenStack Cloud and Beyond with Ansible
  • The Open Open Open Open Cloud
  • OpenStack Security Team Update

Photo credit: Luciot

OpenStack-Ansible networking on CentOS 7 with systemd-networkd

By Leave a Comment

Ethernet switchAlthough OpenStack-Ansible doesn’t fully support CentOS 7 yet, the support is almost ready. I have a four node Ocata cloud deployed on CentOS 7, but I decided to change things around a bit and use systemd-networkd instead of NetworkManager or the old rc scripts.

This post will explain how to configure the network for an OpenStack-Ansible cloud on CentOS 7 with systemd-networkd.

Each one of my OpenStack hosts has four network interfaces and each one has a specific task:

  • enp2s0 – regular network interface, carries inter-host LAN traffic
  • enp3s0 – carries br-mgmt bridge for LXC container communication
  • enp4s0 – carries br-vlan bridge for VM public network connectivity
  • enp5s0 – carries br-vxlan bridge for VM private network connectivity

Adjusting services

First off, we need to get systemd-networkd and systemd-resolved ready to take over networking:

LAN interface

My enp2s0 network interface carries traffic between hosts and handles regular internal LAN traffic.

This one is quite simple, but the rest get a little more complicated.

Management bridge

The management bridge (br-mgmt) carries traffic between LXC containers. We start by creating the bridge device itself:

Now we configure the network on the bridge (I use OpenStack-Ansible’s defaults here):

I run the management network on VLAN 10, so I need a network device and network configuration for the VLAN as well. This step adds the br-mgmt bridge to the VLAN 10 interface:

Finally, we add the VLAN 10 interface to enp3s0 to tie it all together:

Public instance connectivity

My router offers up a few different VLANs for OpenStack instances to use for their public networks. We start by creating a br-vlan network device and its configuration:

We can add this bridge onto the enp4s0 physical interface:

VXLAN private instance connectivity

This step is similar to the previous one. We start by defining our br-vxlan bridge:

My VXLAN traffic runs over VLAN 11, so we need to define that VLAN interface:

We can hook this VLAN interface into the enp5s0 interface now:

Checking our work

The cleanest way to apply all of these configurations is to reboot. The Adjusting services step from the beginning of this post will ensure that systemd-networkd and systemd-resolved come up after a reboot.

Run networkctl to get a current status of your network interfaces:

You should have configured in the SETUP column for all of the interfaces you created. Some interfaces will show as degraded because they are missing an IP address (which is intentional for most of these interfaces).

RHEL 7 STIG v1 updates for openstack-ansible-security

By Leave a Comment

OpenStack LogoDISA’s final release of the Red Hat Enterprise Linux (RHEL) 7 Security Technical Implementation Guide (STIG) came out a few weeks ago and it has plenty of improvements and changes. The openstack-ansible-security role has already been updated with these changes.

Quite a few duplicated STIG controls were removed and a few new ones were added. Some of the controls in the pre-release were difficult to implement, especially those that changed parameters for PKI-based authentication.

The biggest challenge overall was the renumbering. The pre-release STIG used an unusual numbering convention: RHEL-07-123456. The final version used the more standardized “V” numbers, such as V-72225. This change required a substantial patch to bring the Ansible role inline with the new STIG release.

All of the role’s documentation is now updated to reflect the new numbering scheme and STIG changes. The key thing to remember is that you’ll need to use --skip-tag with the new STIG numbers if you need to skip certain tasks.

Note: These changes won’t be backported to the stable/ocata branch, so you need to use the master branch to get these changes.

Have feedback? Found a bug? Let us know!

Takeaways from Bruce Schneier’s talk: “Security and Privacy in a Hyper-connected World”

By Leave a Comment

IBM Interconnect 2017 Bruce SchneierBruce Schneier is one of my favorite speakers when it comes to the topic of all things security. His talk from IBM Interconnect 2017, “Security and Privacy in a Hyper-connected World“, covered a wide range of security concerns.

There were plenty of great quotes from the talk (scroll to the end for those) and I will summarize the main takeaways in this post.

People, process, and technology

Bruce hits this topic a lot and for good reason: a weak link in any of the three could lead to a breach and a loss of data. He talked about the concept of security as a product and a process. Security is part of every product we consume. Whether it’s the safety of the food that makes it into our homes or the new internet-connected thermostat on the wall, security is part of the product.

The companies that sell these products have a wide variety of strategies for managing security issues. Vulnerabilities in an internet-connected teapot are not worth much since there isn’t a lot of value there. It’s probably safe to assume that a teapot will have many more vulnerabilities than your average Apple or Android mobile device. Vulnerabilities in those devices are extremely valuable because the data we carry on those devices is valuable.

Certainty vs. uncertainty

The talk moved into incident response and how to be successful when the worst happens. Automation only works when there’s a high degree of certainty in the situation. If there are variables that can be plugged into an algorithm and a result comes out the other end, automation is fantastic.

Bruce recommended using orchestration when tackling uncertain situations, such as security incident responses. Orchestration involves people following processes and using technology where it makes sense.

He talked about going through TSA checkpoints where metal detectors and x-ray scanners essentially run the show. Humans are around when these pieces of technology detect a problem. If you put a weapon into your carry on, the x-ray scanner will notify a human and that human can take an appropriate response to escalate the problem. If a regular passenger has a firearm in a carry-on bag, the police should be alerted. If an Air Marshal has one, then the situation is handled entirely differently — by a human.

One other aspect he noted was around the uncertainty surrounding our data. Our control over our data, and our control over the systems that hold our data, is decreasing. Bruce remarked that he has more control over what his laptop does than his thermostat.

OODA loop

Bruce raised awareness around the OODA loop and its value when dealing with security incidents. Savvy readers will remember that the OODA loop was the crux of my “Be an inspiration, not an impostor” talk about impostor syndrome.

His point was that the OODA loop is a great way to structure a response during a stressful situation. When the orchestration works well, the defenders can complete an OODA loop faster than their adversaries can. When it works really well, the defenders can find ways to disrupt the adversaries’ OODA loops and thwart the attack.

[Read more…]