2018-03-07
Walt Disney said it best:
We keep moving forward, opening new doors, and doing new things, because we’re curious and curiosity keeps leading us down new paths.
The world of technology is all about change. We tear down the old things that get in our way and we build new technology that takes us to new heights. Tearing down these old things can often be difficult and that forces us to make difficult choices.
Rackspace has been a great home for me for over 11 years. I’ve made the incredibly difficult choice to leave Rackspace on March 9th to pursue new challenges.
Humble beginnings
I came to Rackspace as an entry-level Linux administrator and was amazed by the culture generated by Rackers. The dedication to customers, technology, and quality was palpable from the first few minutes I spent with my team. Although I didn’t know it at the time, I had landed at the epicenter of a sink-or-swim technology learning experience. My team had some very demanding customers with complex infrastructures and it forced me to take plenty of notes (and hard knocks). My manager and teammates supported me through it all.
From there, I served in several different roles. I was a manager of technicians on a support team and had the opportunity to learn how to mentor. One of my favorite leaders said that “good managers know when to put their arm around to people and when to put a boot in their rear.” I reluctantly learned how to do both and I watched my people grow into senior engineers and great leaders.
Datapoint office closing in 2011
I was pulled to Mosso, Rackspace’s first cloud offering, shortly after that and discovered an entirely new world. Rackers force-fed me “Why’s (Poignant) Guide to Ruby” and I started building scripts and web front-ends for various services. Rackspace acquired Slicehost after that and I jumped at the chance to work as an operations engineer on the new infrastructure. That led to a lot of late nights diagnosing problems with Xen hypervisors and rails applications. I met some amazing people and began to realize that St. Louis has some pretty good barbecue (but Texas still has them beat).
Slicehost humor in 2009
Not long after that, I found myself managing an operations team that cared for Slicehost’s infrastructure and Rackspace’s growing Cloud Servers infrastructure. OpenStack appeared later and I jumped at the chance to do operations there. It was an extremely rough experience in the Diablo release, but it taught me a lot. My start with OpenStack involved fixing lots of broken Keystone tests that didn’t run on Python 2.6.
Working on OpenStack in 2012
If you’ve attended some of my talks on impostor syndrome, you may know what came next. We had a security issue and I sent some direct feedback to our CSO about how it was handled. I expected to be told to “pack a box” after that, but I was actually asked to lead a security architecture team in the corporate security group. It was definitely a surprise. I accepted and joined the team as Chief Security Architect. My coworkers called it “joining the dark side”, but I did my best to build bridges between security teams and the rest of the company.
Talking at Rackspace::Solve in 2015
This role really challenged me. I had never operated at the Director level before and our team had a ton of work to do. I found myself stumbling (and floundering) fairly often and I leaned on other leaders in the business for advice. This led me to take some courses on critical thinking, accounting, finance, and tough conversations. I’ve never had a role as difficult as this one.
Our cloud team came calling and asked me to come back and help with some critical projects in the public cloud. We worked on some awesome skunkworks projects that could really change the business. Although they didn’t get deployed in one piece, we found ways to take chunks of the work and optimize different areas of our work. An opportunity came up to bring public cloud experience to the private cloud team and I jumped on that one. I discovered the awesome OpenStack-Ansible project and a strong set of Rackers who were dedicated to bringing high-touch service to customers who wanted OpenStack in their own datacenter.
Impostor syndrome talk at the Boston OpenStack Summit in 2017
During this time, I had the opportunity to deliver several conference talks about OpenStack, Fedora, security, and Ansible. My favorite topic was impostor syndrome and I set out on a mission to help people understand it. My first big talk was at the Fedora Flock conference in Rochester in 2015. This led to deep conversations with technical people in conference hallways, evening events, and even airport terminals about how impostor syndrome affects them. I took those conversations and refined my message several times over.
Talking about impostor syndrome at Fedora Flock 2015 (Photo credit: Kushal Das)
Gratitude
I couldn’t even begin to name a list of Rackers who have helped me along the way. I wouldn’t be where I am now without the help of hundreds of Rackers. They’ve taught me how to build technology, how to navigate a business, and how to be a better human. They have made me who I am today and I’m eternally grateful. I’ve had an incredible amount of hugs this week at the office and I’ve tried my best not to get a face full of tears in the process.
I’d also like to thank all of the people who have allowed me to mentor them and teach them something along the way. One of the best ways to understand something is to teach it to someone else. I relish any opportunity to help someone avoid a mistake I made, or at least be able to throw something soft under them to catch their fall. These people put up with my thick Texas accent, my erratic whiteboard diagrams, and worse of all, my dad jokes.
Another big “thank you” goes out to all of the members of the open source communities who have mentored me and dealt with my patches.
The first big community I joined was the Fedora Linux community. I’ve been fortunate to serve on the board and participate in different working groups. Everyone has been helpful and accommodating, even when I pushed broken package builds. I plan to keep working in the community as long as they will have me!
The OpenStack community has been like family. Everyone - from developers to foundation leaders - has truly been a treat to work with over several years. My work on Rackspace’s public and private clouds has pushed me into various projects within the OpenStack ecosystem and I’ve found everyone to be responsive. OpenStack events are truly inspiring and it is incredible to see so many people from so many places who dedicate themselves to the software and the people that make cloud infrastructure work.
The next adventure
I plan to talk more on this later, but I will be working from home on some projects that are entirely different from what I’m working on now. That adventure starts on March 19th after a week of “funemployment.” I’m incredibly excited about the new opportunity and I’ll share more details when I can.
Top photo credit: Wikipedia
2018-02-28
If you’re on the latest Fedora release, you’re already running lots of modern packages. However, there are those times when you may want to help with testing efforts or try out a new feature in a newer package.
Most of my systems have the updates-testing repository enabled in one way or another. This repository contains packages that package maintainers have submitted to become the next stable package in Fedora. For example, if there is a bug fix for nginx, the package maintainer submits the changes and publish a release. That release goes into the testing repositories and must sit for a waiting period or receive sufficient karma (“works for me” responses) to move into stable repositories.
Getting started
One of the easiest ways to get started is to allow a small amount of packages to be installed from the testing repository on a regular basis. Fully enabling the testing repository for all packages can lead to trouble on occasion, especially if a package maintainer discovers a problem and submits a new testing package.
To get started, open /etc/yum.repos.d/fedora-updates-testing.repo in your favorite text editor (using sudo). This file tells yum and dnf where it should look for packages. The stock testing repository configuration looks like this:
[updates-testing]
name=Fedora $releasever - $basearch - Test Updates
failovermethod=priority
#baseurl=http://download.fedoraproject.org/pub/fedora/linux/updates/testing/$releasever/$basearch/
metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-testing-f$releasever&arch=$basearch
enabled=0
repo_gpgcheck=0
type=rpm
gpgcheck=1
metadata_expire=6h
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
skip_if_unavailable=False
By default, the repository is not enabled (enabled=0).
In this example, let’s consider a situation where you want to test the latest kernel packages as soon as they reach the testing repository. We need to make two edits to the repository configuration:
enabled=1 - Allow yum/dnf to use the repositoryincludepkgs=kernel* - Only allow packages matching kernel* to be installed from the testing repository
The repository configuration should now look like this:
[updates-testing]
name=Fedora $releasever - $basearch - Test Updates
failovermethod=priority
#baseurl=http://download.fedoraproject.org/pub/fedora/linux/updates/testing/$releasever/$basearch/
metalink=https://mirrors.fedoraproject.org/metalink?repo=updates-testing-f$releasever&arch=$basearch
enabled=1
repo_gpgcheck=0
type=rpm
gpgcheck=1
metadata_expire=6h
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-$releasever-$basearch
skip_if_unavailable=False
includepkgs=kernel*
Getting testing packages
Running dnf upgrade kernel* should now pull a kernel from the updates-testing repository. You can verify this by checking the Repository column in the dnf output.
If you feel more adventurous later, you can add additional packages (separated by spaces) to the includepkgs line. The truly adventurous users can leave the repo enabled but remove includepkgs altogether. This will pull all available packages from the testing repository as soon as they are available.
Package maintainers need feedback!
One final note: package maintainers need your feedback on packages. Positive or negative feedback is very helpful. You can search for the package on Bodhi and submit feedback there, or use the fedora-easy-karma script via the fedora-easy-karma package. The script will look through your installed package list and query you for feedback on each one.
Submitting lots of feedback can earn you some awesome Fedora Badges!
Photo credit: US Air Force
2018-01-06
The Overland Expo in Asheville last year was a great event, and one of my favorite sessions covered the basics about radio communications while overlanding. The instructors shared their radios with us and taught us some tips and tricks for how to save power and communicate effectively on the trail.
Back at the office, I was surprised to discover how many of my coworkers had an FCC license already. They gave me tips on getting started and how to learn the material for the exam. I took some of my questions to Twitter and had plenty of help pouring in quickly.
This post covers how I studied, what the exam was like, and what I’ve learned after getting on the air.
The basics
FCC licenses in the US for amateur radio operators have multiple levels. Everything starts with the Technician level and you get the most basic access to radio frequencies. From there, you can upgrade (with another exam) to General, and Extra. Each license upgrade opens up more frequencies and privileges.
Studying
A coworker recommended the official ARRL book for the Technician exam and I picked up a paper copy. The content is extremely dry. It was difficult to remain focused for long periods.
The entire exam is available in the public domain, so you can actually go straight to the questions that you’ll see on the exam and study those. I flipped to the question section in the ARRL book and found the questions I could answer easily (mostly about circuits and electrical parts). For each one that was new or difficult, I flipped back in the ARRL book to the discussion in each chapter and learned the material.
I also used HamStudy.org to quickly practice and keep track of my progress. The site has some handy graphs that show you how many questions you’ve seen and what your knowledge level of different topics really is. I kept working through questions on the site until I was regularly getting 90% or higher on the practice tests.
Testing
Before you test, be sure to get a FCC Registration Number (commonly called a FRN). They are free to get and it ensures that you get your license (often called your ‘ticket’) as soon as possible. I was told that some examiners won’t offer you a test if you don’t have your FRN already.
The next step is to find an amateur radio exam in your area. Exams are available in the San Antonio area every weekend and they are held by different groups. I took mine with the Radio Operators of South Texas and the examiners were great! Some examiners require you to check in with them so they know you are coming to test, but it’s a good idea to do this anyway. Ask how they want to be paid (cash, check, etc), too.
Be sure to take a couple of pencils, a basic calculator, your government issued ID, your payment, and your FRN to the exam. I forgot the calculator but the examiners had a few extras. The examiners complete some paperwork before your exam, and you select one of the available test versions. Each test contains a randomly selected set of 35 questions from the pool of 350.
Go through the test, carefully read each question, and fill in the answer sheet. Three examiners will grade it when you turn it in, and they will fill out your Certificate of Successful Completion of Examination (CSCE). Hold onto this paper just in case something happens with your FCC paperwork.
The examiners will send your paperwork to the FCC and you should receive a license within two weeks. Mine took about 11-12 business days, but I took it just before Thanksgiving. The FCC will send you a generic email stating that there is a new license available and you can download it directly from the FCC’s website.
Lessons learned on the air
Once I passed the exam and keyed up for the first transmission, I feared a procedural misstep more than anything. What if I say my callsign incorrectly? What if I’m transmitting at a power level that is too high? What power level is too high? What am I doing?!
Everyone has to start somewhere and you’re going to make mistakes. Almost 99.9% of my radio contacts so far have been friendly, forgiving, and patient. I’ve learned a lot from listening to other people and from the feedback I get from radio contacts. Nobody will yell at you for using a repeater when simplex should work. Nobody will yell at you if you blast a repeater with 50 watts when 5 would be fine.
I’m on VHF most often and I’ve found many local repeaters on RepeaterBook. Most of the repeaters in the San Antonio area are busiest during commute times (morning and afternoon) as well as lunchtime. I’ve announced my callsign when the repeater has been quiet for a while and often another radio operator will call back. It’s a good idea to mention that you’re new to amateur radio since that will make it easier for others to accept your mistakes and provide feedback.
when I’m traveling long distances, I monitor the national simplex calling frequency (146.520). That’s the CB equivalent of channel 19 where you can announce yourself and have conversations. In busy urban areas, it’s best to work out another frequency with your contact to keep the calling frequency clear.
My equipment
My first purchase was a (cheap) BTECH UV-5X3. The price is fantastic, but the interface is rough to use. Editing saved channels is nearly impossible and navigating the menus requires a good manual to decipher the options. The manual that comes with it is surprisingly brief. There are some helpful how-to guides from other radio operators on various blogs that can help.
I picked up a Kenwood TM-D710G mobile radio from a coworker and mounted it in the car. I wired it up with Anderson Powerpole connectors and that makes things incredibly easy (and portable). The interface on the Kenwood is light years ahead of the BTECH, but the price is 10x more.
My car has the Comet SBB-5NMO antenna mounted with a Comet CP-5NMO lip mount. It fits well on the rear of the 4Runner.
Managing a lot of repeater frequencies is challenging with both radios (exponentially more so with the BTECH), but the open source CHIRP software works well. I installed it on my Fedora laptop and could manage both radios easily. The BTECH radio requires you to download the entire current configuration, edit it, and upload it to the radio. The Kenwood allows you to make adjustments to the radio in real time (which is excellent for testing).
More questions?
If you have more questions about any part of the process, let me know!
2017-12-15
After a recent OpenStack-Ansible (OSA) deployment on CentOS, I found that keepalived was not starting properly at boot time:
Keepalived_vrrp[801]: Cant find interface br-mgmt for vrrp_instance internal !!!
Keepalived_vrrp[801]: Truncating auth_pass to 8 characters
Keepalived_vrrp[801]: VRRP is trying to assign ip address 172.29.236.11/32 to unknown br-mgmt interface !!! go out and fix your conf !!!
Keepalived_vrrp[801]: Cant find interface br-mgmt for vrrp_instance external !!!
Keepalived_vrrp[801]: Truncating auth_pass to 8 characters
Keepalived_vrrp[801]: VRRP is trying to assign ip address 192.168.250.11/32 to unknown br-mgmt interface !!! go out and fix your conf !!!
Keepalived_vrrp[801]: VRRP_Instance(internal) Unknown interface !
systemd[1]: Started LVS and VRRP High Availability Monitor.
Keepalived_vrrp[801]: Stopped
Keepalived[799]: Keepalived_vrrp exited with permanent error CONFIG. Terminating
OSA deployments have a management bridge for traffic between containers. These containers run the OpenStack APIs and other support services. By default, this bridge is called br-mgmt.
The keepalived daemon is starting before NetworkManager can bring up the br-mgmt bridge and that is causing keepalived to fail. We need a way to tell systemd to wait on the network before bringing up keepalived.
Waiting on NetworkManager
There is a special systemd target, network-online.target, that is not reached until all networking is properly configured. NetworkManager comes with a handy service called NetworkManager-wait-online.service that must be complete before the network-online target can be reached:
# rpm -ql NetworkManager | grep network-online
/usr/lib/systemd/system/network-online.target.wants
/usr/lib/systemd/system/network-online.target.wants/NetworkManager-wait-online.service
Start by ensuring that the NetworkManager-wait-online service starts at boot time:
systemctl enable NetworkManager-wait-online.service
Using network-online.target
Next, we tell the keepalived service to wait on network-online.target. Bring up an editor for overriding the keepalived.service unit:
systemctl edit keepalived.service
Once the editor appears, add the following text:
[Unit]
Wants=network-online.target
After=network-online.target
Save the file in the editor and reboot the server. The keepalived service should come up successfully after NetworkManager signals that all of the network devices are online.
Learn more by reading the upstream NetworkTarget documentation.
2017-11-02
The latest release of the Red Hat Enterprise Linux Security Technical Implementation Guide (STIG) was published last week. This release is Version 1, Release 3, and it contains four main changes:
- V-77819 - Multifactor authentication is required for graphical logins
- V-77821 - Datagram Congestion Control Protocol (DCCP) kernel module must be disabled
- V-77823 - Single user mode must require user authentication
- V-77825 - Address space layout randomization (ASLR) must be enabled
Deep dive
Let’s break down this list to understand what each one means.
V-77819 - Multifactor authentication is required for graphical logins
This requirement improves security for graphical logins and extends the existing requirements for multifactor authentication for logins (see V-71965, V-72417, and V-72427). The STIG recommends smartcards (since the US Government often uses CAC cards for multifactor authentication), and this is a good idea for high security systems.
I use Yubikey 4’s as smartcards in most situations and they work anywhere you have available USB slots.
V-77821 - Datagram Congestion Control Protocol (DCCP) kernel module must be disabled
DCCP is often used as a congestion control mechanism for UDP traffic, but it isn’t used that often in modern networks. There have been vulnerabilities in the past that are mitigated by disabling DCCP, so it’s a good idea to disable it unless you have a strong reason for keeping it enabled.
The ansible-hardening role has been updated to disable the DCCP kernel module by default.
V-77823 - Single user mode must require user authentication
Single user mode is often used in emergency situations where the server cannot boot properly or an issue must be repaired without a fully booted server. This mode can only be used at the server’s physical console, serial port, or via out-of-band management (DRAC, iLO, and IPMI). Allowing single-user mode access without authentication is a serious security risk.
Fortunately, every distribution supported by the ansible-hardening role already has authentication requirements for single user mode in place. The ansible-hardening role does not make any adjustments to the single user mode unit file since any untested adjustment could cause a system to have problems booting.
V-77825 - Address space layout randomization (ASLR) must be enabled
ASLR is a handy technology that makes it more difficult for attackers to guess where a particular program is storing data in memory. It’s not perfect, but it certainly raises the difficulty for an attacker. There are multiple settings for this variable and the kernel documentation for sysctl has some brief explanations for each setting (search for randomize_va_space on the page).
Every distribution supported by the ansible-hardening role is already setting kernel.randomize_va_space=2 by default, which applies randomization for the basic parts of process memory (such as shared libraries and the stack) as well as the heap. The ansible-hardening role will ensure that the default setting is maintained.
ansible-hardening is already up to date
If you’re already using the ansible-hardening role’s master branch, these changes are already in place! Try out the new updates and open a bug report if you find any problems.
