Apache’s mod_proxy, mod_ssl, and BitTorrent Sync

Merging railway tracksBitTorrent Sync allows you to keep files synchronized between multiple computers or mobile devices. It’s a handy way to do backups, share files with friends, or automate the movement of data from device to device. It comes with a web frontend, called the Web UI, that allows for connections over HTTP or HTTPS.

Using HTTP across the internet to administer Sync seems totally absurd, so I decided to enable HTTPS. I quickly realized two things:

  • My SSL certificates were now specified in Apache and Sync
  • Sync’s Web UI is relatively slow with SSL enabled (especially over higher latency links)

I really wanted to keep things simple by wedging Sync into my existing Apache configuration using mod_proxy.  That was easier said than done since the Web UI has some hard-coded paths for certain assets, like CSS and Javascript.  After quite a bit of trial end error, this configuration works well:

  ProxyPass /btsync
  ProxyPassReverse /btsync
  ProxyHTMLURLMap /btsync
  Redirect permanent /gui /btsync/gui

The ProxyPass and ProxyPassReverse lines tell Apache where to proxy the requests and it also tells Apache to make requests on behalf of the browser making the request. The ProxyHTMLURLMap directive tells Apache that any requests to /btsync from a client browser should be translated as a request to the root directory (/) of the Web UI. The last line redirects hard-coded requests to /gui up to /btsync/gui instead.

When your configuration is in place, be sure to run a configuration check (httpd -S) and reload the Apache daemon. If you’d like to access your application at a different URI, just replace /btsync in the example configuration with that URI.

Once all this is done, I’m able to access Sync at https://example.com/btsync and Apache handles all of the backend requests properly. On some distributions, you may find that mod_proxy_html isn’t installed by default. You’ll need to install it if you want to use ProxyHTMLURLMap in your configuration. For Fedora users, just install it via yum:

yum install mod_proxy_html

Photo: Old Vintage Railway by Viktor Hanacek

HOWTO: Time Warner Cable and IPv6

IPv6 world launch logoTime Warner has gradually rolled out IPv6 connectivity to their Road Runner customers over the past couple of years and it started appearing on my home network earlier this year.  I had some issues getting the leases to renew properly after they expired (TWC’s default lease length appears to be seven days) and there were some routing problems that cropped up occasionally.  However, over the past month, things seem to have settled down on TWC’s San Antonio network.

Do you have IPv6 yet?

Before you make any adjustments to your network, I’d recommend connecting your computer directly to the cable modem briefly to see if you can get an IPv6 address via stateless autoconfiguration (SLAAC).  You’ll only get one IPv6 address via SLAAC, but we can get a bigger network block later on (keep reading).  Check your computer’s network status to see if you received an IPv6 address.  If you have one, try accessing ipv6.google.com.  You can always check ipv6.icanhazip.com or ipv6.icanhaztraceroute.com as well.

There’s a chance your computer didn’t get an IPv6 address while directly connected to the cable modem.  Here are some possible solutions:

  • Power off the cable modem for 30 seconds, then plug it back in and see if your computer gets an address
  • Ensure you have one of TWC’s approved modems(Bear in mind that not all of these modems support IPv6.)
  • Verify that your computer has IPv6 enabled. (Instructions for Windows, Mac and Linux are available.)

But I want more addresses

If you were able to get an IPv6 address, it’s now time to allocate a network block for yourself and begin using it!  We will request an allocation via DHCPv6.  Every router is a little different, but the overall concept is the same.  Your router will request an allocation on the network and receive that allocation from Time Warner’s network.  From there, your router will assign that block to an interface (most likely your LAN, more on that in a moment) and begin handing our IPv6 addresses to devices in your home.

By default, TWC hands out /64 allocations regardless of what you request via DHCPv6.  I had some success in late 2013 when I requested a /56 but it appears that allocations of that size aren’t available any longer.  Sure, a /64 allocation is gigantic (bigger than the entire IPv4 address space), but getting a /56 would allow you to assign multiple /64 allocations to different interfaces.  Splitting /64’s into smaller subnets is a bad idea.

Let’s talk security

IPv6 eliminates the need for network address translation (NAT).  This means that by the time you finish this howto, each device in your network with have a publicly accessible internet address.  Also, bear in mind that with almost all network devices, firewall rules and ACL’s that are configured with IPv4 will have no effect on IPv6.  This means that you’ll end up with devices on your network with all of their ports exposed to the internet.

In Linux, be sure to use ip6tables (via firewalld, if applicable).  For other network devices, review their firewall configuration settings to see how you can filter IPv6 traffic.  This is a critical step.  Please don’t skip it.

On my Mikrotik device, I have a separate IPv6 firewall interface that I can configure.  Here is my default ruleset:

/ipv6 firewall filter
/ipv6 firewall filter
add chain=input connection-state=related
add chain=input connection-state=established
add chain=forward connection-state=established
add chain=input in-interface=lanbridge
add chain=forward connection-state=related
add chain=input dst-port=546 protocol=udp
add chain=input protocol=icmpv6
add chain=forward protocol=icmpv6
add chain=forward out-interface=ether1-gateway
add action=drop chain=input
add action=drop chain=forward

The first five rules ensure that only related or established connections can make it to my internal LAN. I allow UDP 546 for DHCPv6 connectivity and I’m allowing all ICMPv6 traffic to the router and internal devices. Finally, I allow all of my devices inside the network to talk to the internet and block the remainder of the unmatched traffic.

Configuring the router

It’s no secret that I’m a big fan of Mikrotik devices and I’ll guide you through the setup of IPv6 on the Mikrotik in this post.  Before starting this step, ensure that your firewall is configured (see previous section).

On the Mikrotik, just add a simple DHCPv6 configuration. I’ll call mine ‘twc':

/ipv6 dhcp-client
add add-default-route=yes interface=ether1-gateway pool-name=twc

After that, you should see an allocation pop up within a few seconds (run ipv6 dhcp-client print):

 #    INTERFACE     STATUS        PREFIX                                      EXPIRES-AFTER
 0    ether1-gat... bound         2605:xxxx:xxxx:xxxx::/64                    6d9h15m45s

Check that a new address pool was allocated by running ipv6 pool print:

 #   NAME      PREFIX                                      PREFIX-LENGTH EXPIRES-AFTER
 0 D twc       2605:xxxx:xxxx:xxxx::/64                               64 6d9h13m33s

You can now assign that address pool to an interface. Be sure to assign the block to your LAN interface. In my case, that’s called lanbridge:

/ipv6 address
add address=2605:xxxx:xxxx:xxxx:: from-pool=twc interface=lanbridge

By default, the Mikrotik device will now begin announcing that network allocation on your internal network. Some of your devices may already be picking up IPv6 addresses via SLAAC! Try accessing the Google or icanhazip IPv6 addresses from earlier in the post.

Checking a Linux machine for IPv6 connectivity is easy. Here’s an example from a Fedora 20 server I have at home:

$ ip -6 addr
2: em1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000
    inet6 2605:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx/64 scope global mngtmpaddr dynamic
       valid_lft 2591998sec preferred_lft 604798sec
    inet6 2605:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx/64 scope global deprecated mngtmpaddr dynamic
       valid_lft 1871064sec preferred_lft 0sec

If you only see an address that starts with fe80, that’s your link local address. It’s not an address that can be accessed from the internet.


If you run into some problems or your router can’t pull an allocation via DHCPv6, try the troubleshooting steps from the first section of this post.

Getting assistance from Time Warner is a real challenge. Everyone I’ve contacted via phone or Twitter has not been able to help and many of them don’t even know what IPv6 is. I was even told “we have plenty of regular IPv4 addresses left, don’t worry” when I asked for help. Even my unusual methods haven’t worked:

My old SBG6580 that was issued by Time Warner wouldn’t ever do IPv6 reliably. I ended up buying a SB6121 and I was able to get IPv6 connectivity fairly easily. The SB6121 only does 172mb/sec down — I’ll be upgrading it if TWC MAXX shows up in San Antonio.

Asus Maximus VI Gene – Error 55

It’s been quite a while since I built a computer but I decided to give it a try for a new hypervisor/NAS box at home. I picked up an Asus Maximus VI Gene motherboard since it had some good parts installed and it seems to work well with Linux. This was my first time doing water cooling for the CPU and I picked up a Seidon 240M after getting some recommendations.

Rubber hits the road

Once everything was in the box and the power was applied, I was stuck with an error code. There’s a two-digit LCD display on the motherboard that rapidly flips between different codes during boot-up. If it stays on a code for a while and you don’t get any display output, you have a problem. For me, this Asus Q code was 55.

The manual says it means that RAM isn’t installed. I pulled out my four sticks of RAM and reseated all of them. I still got the same error. After reading a bunch of forum posts, I ran through a lot of troubleshooting steps:

  • Reseat the RAM
  • Try one stick of RAM and add more until the error comes back
  • Reseat the CPU cooler (at least three times)
  • Reseat the CPU (at least three times)
  • Upgrade the BIOS
  • Clear the CMOS
  • Curse loudly, drink a beer, and come back

I still had error 55 and wasn’t going anywhere fast. After some further testing, I found that if I left the two RAM slots next to the CPU empty, the system would boot. If I put any RAM in the two left RAM slots (A1 and A2), the system wouldn’t boot. Here’s an excerpt from the manual:

Asus Maximus VI Gene Motherboard CPU/memory layout

CPU is on the left. RAM slots are A1, A2, B1, B2, left to right.

Fine-tuning the Google search

I adjusted my Google terms to include “A1 A2 slots” and found more posts talking about CPU coolers being installed incorrectly. Mine had to be correct — I installed it four times! I decided to try re-installing it one last time.

When I removed the CPU cooler from the CPU, I noticed something strange. There are four standoffs around the CPU that the cooler would attach to with screws. Those standoffs screwed into posts that connected to a bracket on the back of the motherboard.

Asus motherboard cpu cooler standoff

The lower two standoffs are highlighted.

I removed the two standoffs that were closest to the A1/A2 RAM slots and noticed something peculiar. One side of the standoff had a black coating that seemed a bit tacky while the other side of the standoff was bare metal. Three of the standoffs had the black side down (against the board) while one had the black side up. I unscrewed that standoff and found that the bare metal side was wedged firmly onto some connections that run from the CPU to the A1/A2 RAM slots. Could this be the issue?


After double-checking all of the CPU cooler standoffs and attaching the cooler to the board, I crossed my fingers and hit the power button. The machine shot through POST and I was staring down a Fedora logo that quickly led to a GDM login.

Badly installed cpu cooler standoff

The culprit

I don’t talk about hardware too often on the blog, but I certainly hopes this helps someone else who is desperately trying to find a solution.

Audit RHEL/CentOS 6 security benchmarks with ansible

Ansible logoSecuring critical systems isn’t easy and that’s why security benchmarks exist. Many groups and communities distribute recommendations for securing servers, including NIST, the US Department of Defense (DoD), and the Center for Internet Security (CIS).

Although NIST and DoD are catching up quickly with newer OS releases, I’ve found that the CIS benchmarks are updated very regularly. CIS distributes auditing tools (with paid memberships) that require Java and they’re cumbersome to use, especially on servers where Java isn’t normally installed.

A better way to audit security benchmarks

I set out to create an Ansible playbook that would allow users to audit and (carefully!) remediate servers. The result is on GitHub. Before we go any further, I’d just like to state that I’m not affiliated with CIS in any way and this repository hasn’t been endorsed by CIS. Use it at your own risk.

Getting the playbook onto a machine is easy:

git clone https://github.com/major/cis-rhel-ansible.git

PLEASE review the README and NOTES files in the GitHub repository prior to running the playbook.


Seriously. I mean it. This playbook could knock production environments offline.

The tasks are split into sections (just like the CIS benchmarks themselves) and each section is split into Level 1 and 2 requirements.

Benchmark levels

Level 1 requirements provide good security improvements without a tremendous amount of intrusion into production workloads. With that said, they can still cause issues.

Level 2 requirements provide stronger security improvements but they can adversely affect production server environments. This is where you find things like SELinux, AIDE (including disabling prelinking), and some kernel tweaks for IPv6.

How to use it

I strongly recommend some dry runs with Ansible’s check mode before trying to modify a production system. Also, you can run the playbook against a freshly-installed system and then deploy your applications on top of it. Find out what breaks and disable certain benchmarks that get in the way.

The entire playbook takes less than a minute to run locally on a Rackspace Performance Cloud Server. Your results may vary over remote ssh connections, but I was seeing the playbooks complete over ssh within three to four minutes.

You can also review the variables file to find all the knobs you need to get more aggressive in your audits. If you spot something potentially destructive that needs a variable added, let me know (or submit a pull request).

It’s open source

The entire repository is licensed under Apache License 2.0, so please feel free to submit issues, pull requests, or patches.

Start Jenkins on Fedora 20

Installing Jenkins on Fedora 20 is quite easy thanks to the available Red Hat packages, but I ran into problems when I tried to start Jenkins. Here are the installation steps I followed:

wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
rpm --import http://pkg.jenkins-ci.org/redhat/jenkins-ci.org.key
yum -y install jenkins
systemctl enable jenkins
systemctl start jenkins

Your first error will show up if Java isn’t installed. You can fix that by installing Java:

yum -y install java-1.7.0-openjdk-headless

After installing Java, Jenkins still refused to start. Nothing showed up in the command line or via journalctl -xn, so I jumped into the Jenkins log file (found at /var/log/jenkins/jenkins.log):

Aug 13, 2014 2:21:44 PM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: jetty-8.y.z-SNAPSHOT
Aug 13, 2014 2:21:46 PM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: NO JSP Support for , did not find org.apache.jasper.servlet.JspServlet

My Java knowledge is relatively limited, so I tossed the JSP error message into Google. A stackoverflow thread was the first result and it talked about a possible misconfiguration with Jetty. I tried their trick of using the OPTIONS environment variable, but that didn’t work.

Then I realized that there wasn’t a Jetty package installed on my server. Ouch. The installation continues:

yum -y install jetty-jsp

Jenkins could now get off the ground and I saw the familiar log messages that I’m more accustomed to seeing:

Aug 13, 2014 2:24:26 PM hudson.WebAppMain$3 run
INFO: Jenkins is fully up and running

Much of these problems could stem from the fact that Jenkins RPM’s are built to suit a wide array of system versions and the dependencies aren’t configured correctly. My hope is that the Jenkins project for Fedora 21 will alleviate some of these problems and give the user a better experience.