Very slow ssh logins on Fedora 22

I’ve recently set up a Fedora 22 firewall/router at home (more on that later) and I noticed that remote ssh logins were extremely slow. In addition, sudo commands seemed to stall out for the same amount of time (about 25-30 seconds).

I’ve done all the basic troubleshooting already:

  • Switch to UseDNS no in /etc/ssh/sshd_config
  • Set GSSAPIAuthentication no in /etc/ssh/sshd_config
  • Tested DNS resolution

These lines kept cropping up in my system journal when I tried to access the server using ssh:

dbus[4865]: [system] Failed to activate service 'org.freedesktop.login1': timed out
sshd[7391]: pam_systemd(sshd:session): Failed to create session: Activation of org.freedesktop.login1 timed out
sshd[7388]: pam_systemd(sshd:session): Failed to create session: Activation of org.freedesktop.login1 timed out

The process list on the server looked fine. I could see dbus-daemon and systemd-logind processes and they were in good states. However, it looked like dbus-daemon had restarted at some point and systemd-logind had not been restarted since then. I crossed my fingers and bounced systemd-logind:

systemctl restart systemd-logind

Success! Logins via ssh and escalations with sudo worked instantly.

Restoring wireless and Bluetooth state after reboot in Fedora 22

Thinkpad X1 Carbon 3rd genMy upgrade to Fedora 22 on the ThinkPad X1 Carbon was fairly uneventful and the hiccups were minor. One of the more annoying items that I’ve been struggling with for quite some time is how to boot up with the wireless LAN and Bluetooth disabled by default. Restoring wireless and Bluetooth state between reboots is normally handled quite well in Fedora.

In Fedora 21, NetworkManager saved my settings between reboots. For example, if I shut down with wifi off and Bluetooth on, the laptop would boot up later with wifi off and Bluetooth on. This wasn’t working well in Fedora 22: both the wifi and Bluetooth were always enabled by default.

Digging into rfkill

I remembered rfkill and began testing out some commands. It detected that I had disabled both devices via NetworkManager (soft):

$ rfkill list
0: tpacpi_bluetooth_sw: Bluetooth
    Soft blocked: yes
    Hard blocked: no
2: phy0: Wireless LAN
    Soft blocked: yes
    Hard blocked: no

It looked like systemd has some hooks already configured to manage rfkill via the systemd-rfkill service. However, something strange happened when I tried to start the service:

# systemctl start systemd-rfkill@0
Failed to start systemd-rfkill@0.service: Unit systemd-rfkill@0.service is masked.

Well, that’s certainly weird. While looking into why it’s masked, I found an empty file in /etc/systemd:

# ls -al /etc/systemd/system/systemd-rfkill@.service 
-rwxr-xr-x. 1 root root 0 May 11 16:36 /etc/systemd/system/systemd-rfkill@.service

I don’t remember making that file. Did something else put it there?

# rpm -qf /etc/systemd/system/systemd-rfkill@.service
tlp-0.7-4.fc22.noarch

Ah, tlp!

Configuring tlp

I looked in tlp’s configuration file in /etc/default/tlp and found a few helpful configuration items:

# Restore radio device state (Bluetooth, WiFi, WWAN) from previous shutdown
# on system startup: 0=disable, 1=enable.
# Hint: the parameters DEVICES_TO_DISABLE/ENABLE_ON_STARTUP/SHUTDOWN below
#   are ignored when this is enabled!
RESTORE_DEVICE_STATE_ON_STARTUP=0
 
# Radio devices to disable on startup: bluetooth, wifi, wwan.
# Separate multiple devices with spaces.
#DEVICES_TO_DISABLE_ON_STARTUP="bluetooth wifi wwan"
 
# Radio devices to enable on startup: bluetooth, wifi, wwan.
# Separate multiple devices with spaces.
#DEVICES_TO_ENABLE_ON_STARTUP="wifi"
 
# Radio devices to disable on shutdown: bluetooth, wifi, wwan
# (workaround for devices that are blocking shutdown).
#DEVICES_TO_DISABLE_ON_SHUTDOWN="bluetooth wifi wwan"
 
# Radio devices to enable on shutdown: bluetooth, wifi, wwan
# (to prevent other operating systems from missing radios).
#DEVICES_TO_ENABLE_ON_SHUTDOWN="wwan"
 
# Radio devices to enable on AC: bluetooth, wifi, wwan
#DEVICES_TO_ENABLE_ON_AC="bluetooth wifi wwan"
 
# Radio devices to disable on battery: bluetooth, wifi, wwan
#DEVICES_TO_DISABLE_ON_BAT="bluetooth wifi wwan"
 
# Radio devices to disable on battery when not in use (not connected):
# bluetooth, wifi, wwan
#DEVICES_TO_DISABLE_ON_BAT_NOT_IN_USE="bluetooth wifi wwan"

So tlp’s default configuration doesn’t restore device state and it masked systemd’s rfkill service. I adjusted one line in tlp’s configuration and rebooted:

DEVICES_TO_DISABLE_ON_STARTUP="bluetooth wifi wwan"

After the reboot, both the wifi and Bluetooth functionality were shut off! That’s exactly what I needed.

Extra credit

Thanks to a coworker, I was able to make a NetworkManager script to automatically shut off the wireless LAN whenever I connected to a network via ethernet. This is typically what I do when coming back from an in-person meeting to my desk (where I have ethernet connectivity).

If you want the same automation, just drop this script into /etc/NetworkManager/dispatcher.d/70-wifi-wired-exclusive.sh and make it executable:

#!/bin/bash
export LC_ALL=C
 
enable_disable_wifi ()
{
        result=$(nmcli dev | grep "ethernet" | grep -w "connected")
        if [ -n "$result" ]; then
                nmcli radio wifi off
        fi
}
 
if [ "$2" = "up" ]; then
        enable_disable_wifi
fi

Unplug the ethernet connection, start wifi, and then plug the ethernet connection back in. Once NetworkManager fully connects (DHCP lease obtained, connectivity check passes), the wireless LAN should shut off automatically.

Making things more super with supernova 2.0

OpenStackLogo supernovaI started supernova a little over three years ago with the idea of making it easier to use novaclient. Three years and a few downloads later, it manages multiple different OpenStack clients, like nova, glance, and trove along with some handy features for users who manage a large number of environments.

What’s new?

With some help from some friends who are much better at writing Python than I am (thanks Paul, Matt and Jason), I restructured supernova to make it more testable. The big, awkward SuperNova class was dropped and there are fewer circular imports. In addition, I migrated the cli management components to use the click module. It’s now compatible with Python versions 2.6, 2.7, 3.3 and 3.4.

The overall functionality hasn’t changed much, but there’s a new option to specify a custom supernova configuration that sits in a non-standard location or with a filename other than .supernova. Simply use the -c flag:

supernova -c ~/work/.supernova dfw list
supernova -c ~/personal/supernova-config-v1 staging list

The testing is done with Travis-CI and code coverage is checked with Codecov. Pull requests will automatically be checked with unit tests and I’ll do my best to urge committers to keep test coverage at 100%.

Updating supernova

Version 2.0.0 is already in PyPi, so an upgrade using pip is quite easy:

pip install -U supernova

Aruba access points, EAP, and wpa_supplicant 2.4 bugs

I stumbled upon a strange bug at work one day and found I couldn’t connect to our wireless access points any longer. After some investigation in the systemd journal, I found that my card associated with the access point but never went any further past that. It looked as if the authentication wasn’t ever taking place.

A quick dig through my recent dnf update history didn’t reveal much but then I found a tip from a coworker on an internal wiki that wpa_supplicant 2.4 has problems with certain Aruba wireless access points.

There’s an open ticket on the Red Hat Bugzilla about the issues in wpa_supplicant 2.4. The changelog for 2.4 is lengthy and it has plenty of mentions of EAP; Aruba’s preferred protocol on certain networks. One of those changes could be related. A formal support case is open with Aruba as well.

If this bug affects you, you can return to wpa_supplicant-2.3-3.fc22.x86_64 easily by running:

dnf downgrade wpa_supplicant

This isn’t a good long-term solution, but it fixes the bug and gets you back online.

Allow new windows to steal focus in GNOME 3

GNOME 3 generally works well for me but it has some quirks. One of those quirks is that new windows don’t actually pop up on the screen with focus as they do in Windows and OS X. When opening a new window, you get a “[Windowname] is ready” notification:

GNOME 3 Hangouts is ready

My preference is for new windows to pop in front and steal focus. I can see why that’s not the default since it might cause you to type something in another window where you weren’t expecting to. Fortunately, you can enable what GNOME calls strict window focus with a quick trip to dconf-editor.

Installing dconf-editor is easy:

# RHEL/CentOS 7 and Fedora 21
yum -y install dconf-editor
# Fedora 22
dnf -y install dconf-editor

Open dconf-editor and navigate to org -> gnome -> desktop -> wm -> preferences.

Once you’re there, look for focus-new-windows. The default setting is smart which will keep new windows in the background and alert you via a notification. If you click on smart, a drop down will appear and you can select strict. That will enable functionality similar to OS X and Windows where new windows will pop up in the front and steal your focus.

The new setting takes effect immediately and there’s no need to logout or close and reopen windows.

UPDATE: If you’d like to avoid installing dconf-editor, use Alexander’s suggestion below and simply run:

gsettings set org.gnome.desktop.wm.preferences focus-new-windows 'strict'