Eight years at Rackspace

Rackspace Datapoint office sign

Saying farewell to the Datapoint office location in 2011. That’s where it all started for me in 2006.

Today marks my eight year anniversary at Rackspace and I’m truly honored to work for such a rapidly evolving company that takes the art of customer service to the next level. I continue to learn so much from the community of Rackers around me and I’m glad to have the opportunity to teach them something new as well.

Try out LXC with an Ansible playbook

Ansible logoThe world of containers is constantly evolving lately. The latest turn of events involves the CoreOS developers when they announced Rocket as an alternative to Docker. However, LXC still lingers as a very simple path to begin using containers.

When I talk to people about LXC, I often hear people talk about how difficult it is to get started with LXC. After all, Docker provides an easy-to-use image downloading function that allows you to spin up multiple different operating systems in Docker containers within a few minutes. It also comes with a daemon to help you manage your images and your containers.

Managing LXC containers using the basic LXC tools isn’t terribly easy — I’ll give you that. However, managing LXC through libvirt makes the process much easier. I wrote a little about this earlier in the year.

I decided to turn the LXC container deployment process into an Ansible playbook that you can use to automatically spawn an LXC container on any server or virtual machine. At the moment, only Fedora 20 and 21 are supported. I plan to add CentOS 7 and Debian support soon.

Clone the repository to get started:

git clone https://github.com/major/ansible-lxc.git
cd ansible-lxc

If you’re running the playbook on the actual server or virtual machine where you want to run LXC, there’s no need to alter the hosts file. You will need to adjust it if you’re running your playbook from a remote machine.

As the playbook runs, it will install all of the necessary packages and begin assembling a Fedora 21 chroot. It will register the container with libvirt and do some basic configuration of the chroot so that it will work as a container. You’ll end up with a running Fedora 21 LXC container that is using the built-in default NAT network created by libvirt. The playbook will print out the IP address of the container at the end. The default password for root is fedora. I wouldn’t recommend leaving that for a production use container. ;)

All of the normal virsh commands should work on the container. For example:

# Stop the container gracefully
virsh shutdown fedora21
# Start the container
virsh start fedora21

Feel free to install the virt-manager tool and manage everything via a GUI locally or via X forwarding:

yum -y install virt-manager dejavu* xorg-x11-xauth
# OPTIONAL: For a better looking virt-manager interface, install these, too
yum -y install gnome-icon-theme gnome-themes-standard

Install sysstat on Fedora 21

One of the first tools I learned about after working with Red Hat was sysstat. It can write down historical records about your server at regular intervals. This can help you diagnose CPU usage, RAM usage, or network usage problems. In addition, sysstat also provides some handy command line utilities like vmstat, iostat, and pidstat that give you a live view of what your system is doing.

On Debian-based systems (including Ubuntu), you install the sysstat package and enable it with a quick edit to /etc/default/sysstat and the cron job takes it from there. CentOS and Fedora systems call the collector process using a cron job in /etc/cron.d and it’s enabled by default.

Fedora 21 comes with sysstat 11 and there are now systemd unit files to control the collection and management of stats. You can find the unit files by listing the files in the sysstat RPM:

$ rpm -ql sysstat | grep systemd
/usr/lib/systemd/system/sysstat-collect.service
/usr/lib/systemd/system/sysstat-collect.timer
/usr/lib/systemd/system/sysstat-summary.service
/usr/lib/systemd/system/sysstat-summary.timer
/usr/lib/systemd/system/sysstat.service

These services and timers aren’t enabled by default in Fedora 21. If you run sar after installing sysstat, you’ll see something like this:

# sar
Cannot open /var/log/sa/sa12: No such file or directory
Please check if data collecting is enabled

All you need to do is enable and start the main sysstat service:

systemctl enable sysstat
systemctl start sysstat

From there, systemd will automatically call for collection and management of the statistics using its internal timers. Opening up /usr/lib/systemd/system/sysstat-collect.timer reveals the following:

# /usr/lib/systemd/system/sysstat-collect.timer
# (C) 2014 Tomasz Torcz <tomek@pipebreaker.pl>
#
# sysstat-11.0.0 systemd unit file:
#        Activates activity collector every 10 minutes
 
[Unit]
Description=Run system activity accounting tool every 10 minutes
 
[Timer]
OnCalendar=*:00/10
 
[Install]
WantedBy=sysstat.service

The timer unit file ensures that the sysstat-collect.service is called every 10 minutes based on the real time provided by the system clock. (There are other options to set timers based on relative time of when the server booted or when a user logged into the system). The familiar sa1 command appears in /usr/lib/systemd/system/sysstat-collect.service:

# /usr/lib/systemd/system/sysstat-collect.service
# (C) 2014 Tomasz Torcz <tomek@pipebreaker.pl>
#
# sysstat-11.0.0 systemd unit file:
#        Collects system activity data
#        Activated by sysstat-collect.timer unit
 
[Unit]
Description=system activity accounting tool
Documentation=man:sa1(8)
 
[Service]
Type=oneshot
User=root
ExecStart=/usr/lib64/sa/sa1 1 1

Send weechat notifications via Pushover

IRC is my main communication mechanism and I’ve gradually moved from graphical clients, to irssi and then to weechat. Text-based IRC removes quite a few distractions for me and it allows me to get access to my IRC communications from anything that can act as an ssh client.

I wanted a better way to get notifications when people send me messages and I’m away from my desk. Pushover is a great service that will take notification data via an API and blast it out to various devices. Once you configure your account, just install the mobile application on your device and you’re ready to go.

Connecting weechat to Pushover is quite easy thanks to the pushover.pl script. Go into your main weechat console (usually by pressing META/ALT/OPTION-1 on your keyboard) and install it:

/script install pushover.pl

There are quite a few variables to configure. You can get a list of them by typing:

/set plugins.var.perl.push*

You’ll need two pieces of information to configure the plugin:

  • User key: The user key is displayed on your main account page when you login at Pushover.
  • Application key: Click on Register an Application towards the bottom or use this direct link.

Now you’re ready to configure the plugin:

/set plugins.var.perl.pushover.token [YOUR PUSHOVER APPLICATION TOKEN]
/set plugins.var.perl.pushover.user [YOUR USER KEY]
/set plugins.var.perl.pushover.enabled on

You can test it out quickly by using Freenode’s web chat to send yourself a private message from another account.

Trust an IP address with firewalld’s rich rules

Managing firewall rules with iptables can be tricky at times. The rule syntax itself isn’t terribly difficult but you can quickly run into problems if you don’t save your rules to persistent storage after you get your firewall configured. Things can also get out of hand quickly if you run a lot of different tables with jumps scattered through each.

Why FirewallD?

FirewallD’s goal is to make this process a bit easier by adding a daemon to the mix. You can send firewall adjustment requests to the daemon and it handles the iptables syntax for you. It can also write firewall configurations to disk. It’s especially useful on laptops since you can quickly jump between different firewall configurations based on the network you’re using. You might run a different set of firewall rules at a coffee shop than you would run at home.

Adding a trusted IP address to a device running firewalld requires the use of rich rules.

An example

Consider a situation where you have a server and you want to allow unrestricted connectivity to that server from a bastion or from your home internet connection. First off, determine your default zone (which is most likely “public” unless you’ve changed it to something else):

# firewall-cmd --get-default-zone
public

We will use 11.22.33.44 as our example IP address. Let’s add the rich rule:

firewall-cmd --zone=public --add-rich-rule='rule family="ipv4" source address="11.22.33.44" accept'

Let’s break down what we’re asking firewalld to do. We’re asking to allow IPv4 connectivity from 11.22.33.44 to all ports on the server and we’re asking for that rule to be added to the public (default) zone. If you list the contents of your public zone, it should look like this:

# firewall-cmd --list-all --zone=public
public (default, active)
  interfaces: eth0
  sources:
  services: dhcpv6-client mdns ssh
  ports:
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:
	rule family="ipv4" source address="11.22.33.44" accept