words of wisdom from a systems engineer

Install testing kernels in Fedora


If you’re on the latest Fedora release, you’re already running lots of modern packages. However, there are those times when you may want to help with testing efforts or try out a new feature in a newer package.

Most of my systems have the updates-testing repository enabled in one way or another. This repository contains packages that package maintainers have submitted to become the next stable package in Fedora. For example, if there is a bug fix for nginx, the package maintainer submits the changes and publish a release. That release goes into the testing repositories and must sit for a waiting period or receive sufficient karma (“works for me” responses) to move into stable repositories.

Getting started

One of the easiest ways to get started is to allow a small amount of packages to be installed from the testing repository on a regular basis. Fully enabling the testing repository for all packages can lead to trouble on occasion, especially if a package maintainer discovers a problem and submits a new testing package.

To get started, open /etc/yum.repos.d/fedora-updates-testing.repo in your favorite text editor (using sudo). This file tells yum and dnf where it should look for packages. The stock testing repository configuration looks like this:

name=Fedora $releasever - $basearch - Test Updates

By default, the repository is not enabled (enabled=0).

In this example, let’s consider a situation where you want to test the latest kernel packages as soon as they reach the testing repository. We need to make two edits to the repository configuration:

  • enabled=1 - Allow yum/dnf to use the repository
  • includepkgs=kernel* - Only allow packages matching kernel* to be installed from the testing repository

The repository configuration should now look like this:

name=Fedora $releasever - $basearch - Test Updates

Getting testing packages

Running dnf upgrade kernel* should now pull a kernel from the updates-testing repository. You can verify this by checking the Repository column in the dnf output.

If you feel more adventurous later, you can add additional packages (separated by spaces) to the includepkgs line. The truly adventurous users can leave the repo enabled but remove includepkgs altogether. This will pull all available packages from the testing repository as soon as they are available.

Package maintainers need feedback!

One final note: package maintainers need your feedback on packages. Positive or negative feedback is very helpful. You can search for the package on Bodhi and submit feedback there, or use the fedora-easy-karma script via the fedora-easy-karma package. The script will look through your installed package list and query you for feedback on each one.

Submitting lots of feedback can earn you some awesome Fedora Badges!

Photo credit: US Air Force

Takeaways from my foray into amateur radio


The Overland Expo in Asheville last year was a great event, and one of my favorite sessions covered the basics about radio communications while overlanding. The instructors shared their radios with us and taught us some tips and tricks for how to save power and communicate effectively on the trail.

Back at the office, I was surprised to discover how many of my coworkers had an FCC license already. They gave me tips on getting started and how to learn the material for the exam. I took some of my questions to Twitter and had plenty of help pouring in quickly.

This post covers how I studied, what the exam was like, and what I’ve learned after getting on the air.

The basics

FCC licenses in the US for amateur radio operators have multiple levels. Everything starts with the Technician level and you get the most basic access to radio frequencies. From there, you can upgrade (with another exam) to General, and Extra. Each license upgrade opens up more frequencies and privileges.


A coworker recommended the official ARRL book for the Technician exam and I picked up a paper copy. The content is extremely dry. It was difficult to remain focused for long periods.

The entire exam is available in the public domain, so you can actually go straight to the questions that you’ll see on the exam and study those. I flipped to the question section in the ARRL book and found the questions I could answer easily (mostly about circuits and electrical parts). For each one that was new or difficult, I flipped back in the ARRL book to the discussion in each chapter and learned the material.

I also used to quickly practice and keep track of my progress. The site has some handy graphs that show you how many questions you’ve seen and what your knowledge level of different topics really is. I kept working through questions on the site until I was regularly getting 90% or higher on the practice tests.


Before you test, be sure to get a FCC Registration Number (commonly called a FRN). They are free to get and it ensures that you get your license (often called your ‘ticket’) as soon as possible. I was told that some examiners won’t offer you a test if you don’t have your FRN already.

The next step is to find an amateur radio exam in your area. Exams are available in the San Antonio area every weekend and they are held by different groups. I took mine with the Radio Operators of South Texas and the examiners were great! Some examiners require you to check in with them so they know you are coming to test, but it’s a good idea to do this anyway. Ask how they want to be paid (cash, check, etc), too.

Be sure to take a couple of pencils, a basic calculator, your government issued ID, your payment, and your FRN to the exam. I forgot the calculator but the examiners had a few extras. The examiners complete some paperwork before your exam, and you select one of the available test versions. Each test contains a randomly selected set of 35 questions from the pool of 350.

Go through the test, carefully read each question, and fill in the answer sheet. Three examiners will grade it when you turn it in, and they will fill out your Certificate of Successful Completion of Examination (CSCE). Hold onto this paper just in case something happens with your FCC paperwork.

The examiners will send your paperwork to the FCC and you should receive a license within two weeks. Mine took about 11-12 business days, but I took it just before Thanksgiving. The FCC will send you a generic email stating that there is a new license available and you can download it directly from the FCC’s website.

Lessons learned on the air

Once I passed the exam and keyed up for the first transmission, I feared a procedural misstep more than anything. What if I say my callsign incorrectly? What if I’m transmitting at a power level that is too high? What power level is too high? What am I doing?!

Everyone has to start somewhere and you’re going to make mistakes. Almost 99.9% of my radio contacts so far have been friendly, forgiving, and patient. I’ve learned a lot from listening to other people and from the feedback I get from radio contacts. Nobody will yell at you for using a repeater when simplex should work. Nobody will yell at you if you blast a repeater with 50 watts when 5 would be fine.

I’m on VHF most often and I’ve found many local repeaters on RepeaterBook. Most of the repeaters in the San Antonio area are busiest during commute times (morning and afternoon) as well as lunchtime. I’ve announced my callsign when the repeater has been quiet for a while and often another radio operator will call back. It’s a good idea to mention that you’re new to amateur radio since that will make it easier for others to accept your mistakes and provide feedback.

when I’m traveling long distances, I monitor the national simplex calling frequency (146.520). That’s the CB equivalent of channel 19 where you can announce yourself and have conversations. In busy urban areas, it’s best to work out another frequency with your contact to keep the calling frequency clear.

My equipment

My first purchase was a (cheap) BTECH UV-5X3. The price is fantastic, but the interface is rough to use. Editing saved channels is nearly impossible and navigating the menus requires a good manual to decipher the options. The manual that comes with it is surprisingly brief. There are some helpful how-to guides from other radio operators on various blogs that can help.

I picked up a Kenwood TM-D710G mobile radio from a coworker and mounted it in the car. I wired it up with Anderson Powerpole connectors and that makes things incredibly easy (and portable). The interface on the Kenwood is light years ahead of the BTECH, but the price is 10x more.

My car has the Comet SBB-5NMO antenna mounted with a Comet CP-5NMO lip mount. It fits well on the rear of the 4Runner.

Managing a lot of repeater frequencies is challenging with both radios (exponentially more so with the BTECH), but the open source CHIRP software works well. I installed it on my Fedora laptop and could manage both radios easily. The BTECH radio requires you to download the entire current configuration, edit it, and upload it to the radio. The Kenwood allows you to make adjustments to the radio in real time (which is excellent for testing).

More questions?

If you have more questions about any part of the process, let me know!

Ensuring keepalived starts after the network is ready


After a recent OpenStack-Ansible (OSA) deployment on CentOS, I found that keepalived was not starting properly at boot time:

Keepalived_vrrp[801]: Cant find interface br-mgmt for vrrp_instance internal !!!
Keepalived_vrrp[801]: Truncating auth_pass to 8 characters
Keepalived_vrrp[801]: VRRP is trying to assign ip address to unknown br-mgmt interface !!! go out and fix your conf !!!
Keepalived_vrrp[801]: Cant find interface br-mgmt for vrrp_instance external !!!
Keepalived_vrrp[801]: Truncating auth_pass to 8 characters
Keepalived_vrrp[801]: VRRP is trying to assign ip address to unknown br-mgmt interface !!! go out and fix your conf !!!
Keepalived_vrrp[801]: VRRP_Instance(internal) Unknown interface !
systemd[1]: Started LVS and VRRP High Availability Monitor.
Keepalived_vrrp[801]: Stopped
Keepalived[799]: Keepalived_vrrp exited with permanent error CONFIG. Terminating

OSA deployments have a management bridge for traffic between containers. These containers run the OpenStack APIs and other support services. By default, this bridge is called br-mgmt.

The keepalived daemon is starting before NetworkManager can bring up the br-mgmt bridge and that is causing keepalived to fail. We need a way to tell systemd to wait on the network before bringing up keepalived.

Waiting on NetworkManager

There is a special systemd target,, that is not reached until all networking is properly configured. NetworkManager comes with a handy service called NetworkManager-wait-online.service that must be complete before the network-online target can be reached:

# rpm -ql NetworkManager | grep network-online

Start by ensuring that the NetworkManager-wait-online service starts at boot time:

systemctl enable NetworkManager-wait-online.service


Next, we tell the keepalived service to wait on Bring up an editor for overriding the keepalived.service unit:

systemctl edit keepalived.service

Once the editor appears, add the following text:


Save the file in the editor and reboot the server. The keepalived service should come up successfully after NetworkManager signals that all of the network devices are online.

Learn more by reading the upstream NetworkTarget documentation.

Changes in RHEL 7 Security Technical Implementation Guide Version 1, Release 3

ansible-hardening logoThe latest release of the Red Hat Enterprise Linux Security Technical Implementation Guide (STIG) was published last week. This release is Version 1, Release 3, and it contains four main changes:

  • V-77819 - Multifactor authentication is required for graphical logins
  • V-77821 - Datagram Congestion Control Protocol (DCCP) kernel module must be disabled
  • V-77823 - Single user mode must require user authentication
  • V-77825 - Address space layout randomization (ASLR) must be enabled

Deep dive

Let’s break down this list to understand what each one means.

V-77819 - Multifactor authentication is required for graphical logins

This requirement improves security for graphical logins and extends the existing requirements for multifactor authentication for logins (see V-71965, V-72417, and V-72427). The STIG recommends smartcards (since the US Government often uses CAC cards for multifactor authentication), and this is a good idea for high security systems.

I use Yubikey 4’s as smartcards in most situations and they work anywhere you have available USB slots.

V-77821 - Datagram Congestion Control Protocol (DCCP) kernel module must be disabled

DCCP is often used as a congestion control mechanism for UDP traffic, but it isn’t used that often in modern networks. There have been vulnerabilities in the past that are mitigated by disabling DCCP, so it’s a good idea to disable it unless you have a strong reason for keeping it enabled.

The ansible-hardening role has been updated to disable the DCCP kernel module by default.

V-77823 - Single user mode must require user authentication

Single user mode is often used in emergency situations where the server cannot boot properly or an issue must be repaired without a fully booted server. This mode can only be used at the server’s physical console, serial port, or via out-of-band management (DRAC, iLO, and IPMI). Allowing single-user mode access without authentication is a serious security risk.

Fortunately, every distribution supported by the ansible-hardening role already has authentication requirements for single user mode in place. The ansible-hardening role does not make any adjustments to the single user mode unit file since any untested adjustment could cause a system to have problems booting.

V-77825 - Address space layout randomization (ASLR) must be enabled

ASLR is a handy technology that makes it more difficult for attackers to guess where a particular program is storing data in memory. It’s not perfect, but it certainly raises the difficulty for an attacker. There are multiple settings for this variable and the kernel documentation for sysctl has some brief explanations for each setting (search for randomize_va_space on the page).

Every distribution supported by the ansible-hardening role is already setting kernel.randomize_va_space=2 by default, which applies randomization for the basic parts of process memory (such as shared libraries and the stack) as well as the heap. The ansible-hardening role will ensure that the default setting is maintained.

ansible-hardening is already up to date

If you’re already using the ansible-hardening role’s master branch, these changes are already in place! Try out the new updates and open a bug report if you find any problems.

Import RPM repository GPG keys from other keyservers temporarily

Keys, but not gpg keysI’ve been working through some patches to OpenStack-Ansible lately to optimize how we configure yum repositories in our deployments. During that work, I ran into some issues where was returning 500 errors for some requests to retrieve GPG keys.

Ansible was returning this error:

curl: (22) The requested URL returned error: 502 Proxy Error
error: import read failed(2)

How does the rpm command know which keyserver to use? Let’s use the --showrc argument to show how it is configured:

$ rpm --showrc | grep hkp
-14: _hkp_keyserver
-14: _hkp_keyserver_query   %{_hkp_keyserver}:11371/pks/lookup?op=get&search=0x

How do we change this value temporarily to test a GPG key retrieval from a different server? There’s an argument for that as well: --define:

$ rpm --help | grep define
  -D, --define='MACRO EXPR'        define MACRO with value EXPR

We can assemble that on the command line to set a different keyserver temporarily:

# rpm -vv --define="%_hkp_keyserver" --import 0x61E8806C
-- SNIP --
D: adding "63deac79abe7ad80e147d671c2ac5bd1c8b3576e" to Sha1header index.
-- SNIP --

Let’s verify that our new key is in place:

# rpm -qa | grep -i gpg-pubkey-61E8806C
# rpm -qi gpg-pubkey-61e8806c-5581df56
Name        : gpg-pubkey
Version     : 61e8806c
Release     : 5581df56
Architecture: (none)
Install Date: Wed 20 Sep 2017 10:17:11 AM CDT
Group       : Public Keys
Size        : 0
License     : pubkey
Signature   : (none)
Source RPM  : (none)
Build Date  : Wed 17 Jun 2015 03:57:58 PM CDT
Build Host  : localhost
Relocations : (not relocatable)
Packager    : CentOS Virtualization SIG ( <[email protected]>
Summary     : gpg(CentOS Virtualization SIG ( <[email protected]>)
Description :
Version: rpm-4.11.3 (NSS-3)



If you want to override the value permanently, create a ~/.rpmmacros file and add the following line to it:


Photo credit: Wikipedia