Tagged with virtualization

Changing your ssh server’s port from the default: Is it worth it?

Changing my ssh port from the default port (22) has been one of my standard processes for quite some time when I build new servers or virtual machines. However, I see arguments crop up regularly about it (like this reddit thread or this other one).

Before I go any further, let’s settle the “security through obscurity” argument. (This could probably turn into its own post but I’ll be brief for now.) Security should always be applied in layers. This provides multiple levels of protection from initial attacks, like information gathering attempts or casual threats against known vulnerabilities. In addition, these layers of security should be applied within the environment so that breaking into one server after getting a pivot point in the environment should be just as difficult (if not more difficult) than the original attack that created the pivot point. If “security through obscurity” tactics make up one layer of a multi-layered solution, I’d encourage you to obscure your environment as long as it doesn’t affect your availability.

The key takeaway is:

Security through obscurity is effective if it’s one layer in a multi-layer security solution

Let’s get back to the original purpose of the post.

The biggest benefit to changing the port is to avoid being seen by casual scans. The vast majority of people hunting for any open ssh servers will look for port 22. Some will try the usual variants, like 222 and 2222, but those are few and far between. I ran an experiment with a virtual machine exposed to the internet which had sshd listening on port 22. The server stayed online for one week and then I changed the ssh port to 222. The number of attacks dropped by 98%. Even though this is solely empirical evidence, it’s clear that moving off the standard ssh port reduces your server’s profile.

If it’s more difficult to scan for your ssh server, your chances of being attacked with an ssh server exploit are reduced. A determined attacker can still find the port if they know your server’s IP address via another means (perhaps via a website you host) and they can launch attacks once they find it. Paranoid server administrators might want to check into port knocking to reduce that probability even further.

Remembering the non-standard ssh port can be annoying, but if you have a standard set of workstations that you use for access your servers, just utilize your ~/.ssh/config file to specify certain ports for certain servers. For example:

Host *.mycompany.com
  Port 4321
 
Host nonstandard.mypersonalstuff.com
  Port 2345
 
Host *.mypersonalstuff.com
  Port 5432

If you run into SELinux problems with a non-standard ssh port, there are plenty of guides on this topic.. The setroubleshoot-server package helps out with this as well.

# semanage port -a -t ssh_port_t -p tcp 4321
# semanage port -l | grep ssh
ssh_port_t                     tcp      4321,22

Here is my list of ssh lockdown practices when I build a new server:

  • Update the ssh server package and ensure that automatic updates are configured
  • Enable SELinux and allow a non-standard ssh port
  • Add my ssh public key to the server
  • Disable password logins for ssh
  • Adjust my AllowUsers setting in sshd_config to only allow my user
  • Disable root logins
  • For servers with sensitive data, I install fail2ban
Tagged , , , , , , , , , , ,

virt-manager won’t release the mouse when using ssh forwarding from OS X

The latest versions of virt-manager don’t release the mouse pointer when you’re doing X forwarding to a machine running OS X. This can lead to a rather frustrating user experience since your mouse pointer is totally stuck in the window. Although this didn’t affect me with CentOS 6 hosts, Fedora 18 hosts were a problem.

There’s a relatively elegant fix from btm.geek that solved it for me. On your Mac, exit X11/Xquartz and create an ~/.Xmodmap file containing this:

clear Mod1
keycode 66 = Alt_L
keycode 69 = Alt_R
add Mod1 = Alt_L
add Mod1 = Alt_R

Start X11/Xquartz once more and virt-manager should release your mouse pointer if you hold the left control key and left option at the same time.

Tagged , , , , ,

Late night virtualization frustration with kvm

I dragged out an old Aopen MP57-D tonight that was just sitting in the closet and decided to load up kvm on Fedora 18. I soon found myself staring at a very brief error message upon bootup:

kvm: disabled by bios

After a reboot, the BIOS screen was up and I saw that Virtualization and VT-d were both enabled. Trusted execution (TXT) was disabled, so I enabled it for kicks and rebooted. Now I had two errors:

kvm: disable TXT in the BIOS or activate TXT before enabling KVM
kvm: disabled by bios

Time for another trip to the BIOS. I disabled TXT, rebooted, and I was back to the same error where I first started. A quick check of /proc/cpuinfo showed that I had the right processor extensions. Even the output of lshw showed that I should be ready to go. Some digging in Google led me to a blog post for a fix on Dell Optiplex hardware.

The fix was to do this:

  1. Within the BIOS, disable virtualization, VT-d, and TXT
  2. Save the BIOS configuration, reboot, and pull power to the computer at grub
  3. Within the BIOS, enable virtualization and VT-d but leave TXT disabled
  4. Save the BIOS configuration, reboot, and pull power to the computer at grub
  5. Boot up the computer normally

Although it seems a bit archaic, this actually fixed my problem and set me on my way.

Tagged , , , ,

SELinux, Xen, and block devices in Fedora 17

If you try to run Xen without libvirt on Fedora 17 with SELinux in enforcing mode, you’ll be butting heads with SELinux in no time. You’ll probably be staring at something like this:

# xm create -c fedora17
Using config file "/etc/xen/fedora17".
Error: Disk isn't accessible

If you have setroubleshoot and setroubleshoot-server installed, you should have a friendly message in /var/log/messages telling you the source of the problem:

setroubleshoot: SELinux is preventing /usr/bin/python2.7 from read access on the blk_file dm-1. 
For complete SELinux messages. run sealert -l 4d890105-d9a4-4b3e-a674-ba7e952942dc

The Xen daemon (the python process mentioned in the SELinux denial) is running with a context type of xend_t but the block device I’m trying to use for the VM has fixed_disk_device_t:

# ps axZ | grep xend
system_u:system_r:xend_t:s0       953 ?        SLl    0:40 /usr/bin/python /usr/sbin/xend
# ls -alZ /dev/dm-1
brw-rw----. root disk system_u:object_r:fixed_disk_device_t:s0 /dev/dm-1

SELinux isn’t going to allow this to work. However, even if we fix this, SELinux will balk about three additional issues and we’ll need to adjust the contexts on every new fixed block device we make. To get over the hump, change the context type on your block device to xen_image_t and re-run the xm create:

# chcon -t xen_image_t /dev/dm-1
# ls -alZ /dev/dm-1
brw-rw----. root disk system_u:object_r:xen_image_t:s0 /dev/dm-1
# xm create -c fedora17
Using config file "/etc/xen/fedora17".
Error: out of pty devices

You’ll find three new denials in /var/log/messages:

setroubleshoot: SELinux is preventing /usr/bin/python2.7 from read access on the file group.
For complete SELinux messages. run sealert -l b1392df4-dda4-4b82-914c-1e20c62fc898
setroubleshoot: SELinux is preventing /usr/bin/python2.7 from setattr access on the chr_file 1.
For complete SELinux messages. run sealert -l 3e09edc3-aeb7-49f5-96e1-d8148afda48f
setroubleshoot: SELinux is preventing /usr/bin/python2.7 from execute access on the file pt_chown.
For complete SELinux messages. run sealert -l 86395f09-5f33-4f66-8d02-519b61e54139

As much as it pains me to suggest it, you can create a custom module to allow all four of these operations by xend:

# grep xend /var/log/audit/audit.log | audit2allow -M custom_xen
WARNING: Policy would be downgraded from version 27 to 26.
******************** IMPORTANT ***********************
To make this policy package active, execute:
 
semodule -i custom_xen.pp
 
# semodule -i custom_xen.pp

You should now be able to start your VM without any complaints from SELinux. I’ll reiterate that this isn’t ideal, but it’s the best balance of security and convenience that I’ve found so far.

Tagged , , , , ,

Red Hat Summit 2012: Thursday

Thursday has felt like the busiest, most jam-packed day of the week. The morning started off with three keynotes from HP, Intel, and Red Hat’s CTO, Brian Stevens.

HP’s message centered around converged cloud and that customers don’t need an all or nothing solution. They can pull the best pieces from every type of hosting to do what’s best for their business. The presentation from Intel was extremely heavy on the marketing side and didn’t have much to do with Red Hat. Pauline Nist talked about how chip fabs operate, the heavy costs involved, and how their processors have changed over time. It felt a bit more like a sales pitch than anything else. She wrapped up with a pretty interesting time lapse video of the construction of a fab building and how the chips are made within the fab.

Jim Curry talking about OpenStack during the Red Hat Summit 2012 keynoteBrian Stevens talked a lot about keeping things in the open, reducing vendor lock-in, and pushing for innovation from multiple sources. He talked a lot about OpenStack and how it allows people to deliver a consistent user experience on a very open platform.

There was even a video with Rackspace’s own Jim Curry talking about the unique challenges for building a cloud orchestration layer and how OpenStack solves quite a few of those challenges.

The sessions broke out after that and I made a beeline for Dan Walsh’s session entitled “Multi-tenancy Virtualization Challenges and Solutions.” I first ran into Dan at the FUDCon in Tempe and his session there about SELinux changed my mind about it for good. He covered the basics about the dangers of multi-tenant clouds and then went through multiple examples of how to mitigate the risks. Some of the technologies mentioned included sVirt, SELinux policies, and libseccomp.

The discussion about libseccomp caught my attention because the idea is genius. We know that SELinux itself is quite solid. However, what if a user finds some other flaw in the kernel which allows them to circumvent the SELinux layer altogether? The seccomp additions to linux 3.5 allow you specify which syscalls a particular process is allowed to make. Dan gave an example of QEMU when running KVM. If QEMU only needs 20-30 syscalls to get its job done, why allow it to run every syscall available in the x86_64 and x86 instruction sets? With seccomp, you can specify which syscalls are allowed and what you want to have happen when a syscall is requested that isn’t allowed. Even if someone found a good kernel flaw, they might get blocked from doing anything malicious if the process isn’t allowed to make a syscall necessary to tear up the system.

I met up with the Beefy Miracle himself just before lunch for a photo op:

Major and the Beefy Miracle

Major and the Beefy Miracle


Another good session was entitled “Using an Open Source Framework to Catch the Bad Guy” and it covered auditd in more detail than I imagined was even possible. Mark St. Laurent talked about how auditd’s standard configuration will work for most users but that the government requires some pretty hefty adjustments. I wasn’t aware that you can actually tell auditd to halt the server if it runs out of room to write the audit log to the disk. The US Government requires this to be enabled on many machines.

The last session I attended was Sanjay Rao’s “Tuning Red Hat Systems for Databases” and it was tremendous. He talked about the differences between OLTP/DSS workloads and how all kinds of settings affect their performance. He covered everything from power management to disk elevators to NUMA. There were what seemed like a million slides and all of them contained some really good information. I really wish he could have cracked the presentation into two sessions to allow for some more discussion around details. If you can find the slides from this session, be sure to look through them.

Fenway Park Press BoxAs the day ended, a line of buses pulled up in front of the convention center and we were whisked away to Fenway Park for a private party sponsored by IBM. One of my childhood dreams was to travel to Fenway Park (and Wrigley Field) and I can say I’m halfway done! The park was amazing and it was truly an experience to walk up next to the Green Monster and look up to see how tall it really is.

The food was great and there was plenty of Sam Adams on tap. I ran into quite a few Red Hat folks and eventually found Thomas Cameron so I could thank him for the great SELinux session he ran on Wednesday. I’m going to take his slides back and share them at work to inspire some confidence around managing systems with SELinux enabled. A few twitter friends came up and we had some great conversations about cloud computing, OpenStack, and good beer. For anyone who attends class or works at Harvard, be sure to sync up with Philip Durbin. He’s a smart guy and he was a lot of fun to talk to.

Friday’s the last day and then the trip home begins. I’ll try to write up a wrap-up post tomorrow from the airport.

Tagged , , , , , , , , , ,