Tagged with network

Changing your ssh server’s port from the default: Is it worth it?

Changing my ssh port from the default port (22) has been one of my standard processes for quite some time when I build new servers or virtual machines. However, I see arguments crop up regularly about it (like this reddit thread or this other one).

Before I go any further, let’s settle the “security through obscurity” argument. (This could probably turn into its own post but I’ll be brief for now.) Security should always be applied in layers. This provides multiple levels of protection from initial attacks, like information gathering attempts or casual threats against known vulnerabilities. In addition, these layers of security should be applied within the environment so that breaking into one server after getting a pivot point in the environment should be just as difficult (if not more difficult) than the original attack that created the pivot point. If “security through obscurity” tactics make up one layer of a multi-layered solution, I’d encourage you to obscure your environment as long as it doesn’t affect your availability.

The key takeaway is:

Security through obscurity is effective if it’s one layer in a multi-layer security solution

Let’s get back to the original purpose of the post.

The biggest benefit to changing the port is to avoid being seen by casual scans. The vast majority of people hunting for any open ssh servers will look for port 22. Some will try the usual variants, like 222 and 2222, but those are few and far between. I ran an experiment with a virtual machine exposed to the internet which had sshd listening on port 22. The server stayed online for one week and then I changed the ssh port to 222. The number of attacks dropped by 98%. Even though this is solely empirical evidence, it’s clear that moving off the standard ssh port reduces your server’s profile.

If it’s more difficult to scan for your ssh server, your chances of being attacked with an ssh server exploit are reduced. A determined attacker can still find the port if they know your server’s IP address via another means (perhaps via a website you host) and they can launch attacks once they find it. Paranoid server administrators might want to check into port knocking to reduce that probability even further.

Remembering the non-standard ssh port can be annoying, but if you have a standard set of workstations that you use for access your servers, just utilize your ~/.ssh/config file to specify certain ports for certain servers. For example:

Host *.mycompany.com
  Port 4321
 
Host nonstandard.mypersonalstuff.com
  Port 2345
 
Host *.mypersonalstuff.com
  Port 5432

If you run into SELinux problems with a non-standard ssh port, there are plenty of guides on this topic.. The setroubleshoot-server package helps out with this as well.

# semanage port -a -t ssh_port_t -p tcp 4321
# semanage port -l | grep ssh
ssh_port_t                     tcp      4321,22

Here is my list of ssh lockdown practices when I build a new server:

  • Update the ssh server package and ensure that automatic updates are configured
  • Enable SELinux and allow a non-standard ssh port
  • Add my ssh public key to the server
  • Disable password logins for ssh
  • Adjust my AllowUsers setting in sshd_config to only allow my user
  • Disable root logins
  • For servers with sensitive data, I install fail2ban
Tagged , , , , , , , , , , ,

New icanhaz features: reverse DNS and traceroutes

After adding some upgrades for icanhazip.com, I wanted to go a bit further. Adding reverse DNS (PTR) lookups and traceroutes seemed like a decent idea!

Want to beta test some new features on icanhazptr.com and icanhaztrace.com? Give them a try!

Getting your reverse DNS entry is easy:

$ curl -4 icanhazptr.com
ord.icanhazip.com
$ curl -6 icanhazptr.com
ord.icanhazip.com

Traceroutes are straightforward as well:

$ curl -4 icanhaztrace.com
traceroute to 166.78.118.193 (166.78.118.193), 30 hops max, 60 byte packets
 1  212.111.33.229  20.031 ms
 2  212.111.33.233  1.011 ms
 3  149.11.30.61  107.976 ms
...
$ curl -6 icanhaztrace.com
traceroute to 2001:4801:7818:6:abc5:ba2c:ff10:275f (2001:4801:7818:6:abc5:ba2c:ff10:275f), 30 hops max, 80 byte packets
 1  2a01:7e00:ffff:0:8a43:e1ff:fea3:fa7f  2.183 ms
 2  2001:4d78:fe01:2:1:3:b90:1  1.330 ms
 3  2001:978:2:45::d:1  8.388 ms
...

While this sits in beta, here are some things to keep in mind:

  • If a PTR record doesn’t exist for your IP address, your IP address will be returned
  • Failing traceroutes will cause your IP address to be returned
  • A PTR record will be chosen at random if multiple PTR records are returned
  • PTR lookups for traceroutes are currently disabled

Let me know if you find any bugs.

Tagged , , , , , ,

Red Hat Summit 2012: Wednesday

Wednesday was action-packed with dramatic keynotes and great sessions. The morning was kicked off by Paul Cormier and he talked about some new products coming from Red Hat. Much of the product releases were centered around cloud offerings (like Openshift) and his talk was mainly aimed at CIO’s and decision makers. There wasn’t a lot of technical detail within his talk but it was refreshing to hear a Linux vendor talk about their products as being revolutionary steps in pulling away from vendor lock-in and proprietary solutions.

Paul was followed by Irfan Khan who talked about the value of very low latency information exchange and processing. He drove home the point that the biggest value we can gain from information in the current age is related to our ability to gather and interpret the information in as close to real time as possible. I expected a speech from a SAP employee to be relatively dry but I was pleasantly surprised to find that he made a lot of good points. Irfan emphasized that big data providers need to find a way to fit into their customers’ landscape without causing too much disruption while also providing some real benefits.

My first session was Jeff Darcy’s discussion about Red Hat’s storage offering and what GlusterFS advancements were on the horizon. His talk was standing room only and he covered a lot of highly technical points about GlusterFS. I’m getting the feeling that GlusterFS is gaining more momentum and that we’ll be seeing more features around consistency and performance very soon.

As a fan of SELinux, I made sure that I was in Thomas Cameron’s “SELinux for Mere Mortals” class. Although I feel relatively confident that I can solve SELinux problems when I find them, Thomas covered a lot of easier solutions that I hadn’t previously considered. His explanation of the basics of SELinux are a must read for any system administrator working on a Red Hat system. I managed to find his slides from last year but he said the new slides should be up by Friday.

I attended another good class about managing network resources with Red Hat. Although the slides were a little wordy, the content was extremely good. The speaker talked about receiving a 40Gb/sec ethernet card from Mellanox and how he bumped the performance from 8Gb/sec to 37Gb/sec by adjusting CPU pinning (for NUMA) as well as some kernel configuration around TCP buffers. It was an eye-opening discussion and it was a good session for people who are trying to find bottlenecks in their hardware.

The afternoon was spend mingling with GlusterFS developers and users as well as the people working the Fedora booth. I managed to pick up some Fedora stickers but I’ve yet to get my picture taken with the life-size Beefy Miracle hot dog. That’s my goal for tomorrow.

The night wrapped up with the Red Hat Certified Professionals Party at McGreevy’s across from the Hynes Convention Center. I ran into a bunch of fellow RHCA’s and RHCE’s who read my blog and I was glad some of the posts were able to help them along their way to becoming certified. Congratulations to the folks who passed the early rounds of the new JBoss exams! Being the first ones through that process certainly can’t be easy.

For anyone who is working towards their RHCA, be sure to read my post about my experience with it. It’s a long haul, but the knowledge you’ll gain will be worth it.

Tagged , , , , , , , ,

Lesser-known but extremely handy Linux tools

Kristóf Kovács has a fantastic post about some lesser-known Linux tools that can really come in handy in different situations.

If you haven’t tried dstat (I hadn’t until I saw Kristóf’s post), this is a great one to try. You can keep a running tally on various server metrics including load average, network transfer, and disk operations.

Here is some sample output:

----total-cpu-usage---- ---paging-- ---load-avg--- ------memory-usage----- -net/total- ---procs--- --io/total- ---system-- ----tcp-sockets----
usr sys idl wai hiq siq|  in   out | 1m   5m  15m | used  buff  cach  free| recv  send|run blk new| read  writ| int   csw |lis act syn tim clo
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|1314B  180B|  0   0   0|   0     0 |  70    80 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|1779B 1004B|  0   0   0|   0     0 |  84    78 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M| 904B  362B|1.0   0 1.0|   0     0 |  75    86 | 13   9   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  386M|2203B 1559B|  0   0   0|   0     0 | 180   127 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  386M| 260B  130B|  0   0   0|   0     0 |  53    66 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|  52B  114B|  0   0   0|   0     0 |  54    77 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|2271B  872B|  0   0   0|   0     0 |  94    79 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|  52B  130B|  0   0   0|   0     0 |  54    74 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|1126B 1254B|  0   0   0|   0  24.0 |  80    87 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|1030B  130B|  0   0   0|   0     0 |  88    82 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 578B  114B|  0   0   0|   0     0 |  53    64 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|1597B  890B|  0   0   0|   0     0 |  85    79 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 552B  114B|  0   0   0|   0     0 |  63    77 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|1624B 1254B|  0   0   0|   0     0 |  81    75 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 478B  114B|  0   0   0|   0     0 |  67    73 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 418B  114B|  0   0   0|   0     0 |  59    74 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|1265B  874B|  0   0   0|   0     0 |  82    73 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 758B  114B|  0   0   0|   0     0 |  60    80 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|1236B 1255B|  0   0   0|   0  4.00 |  93    79 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|  52B  130B|  0   0   0|   0     0 |  71    70 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M| 214B  114B|  0   0   0|   0     0 |  55    73 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M|1201B  890B|  0   0   0|   0     0 |  80    80 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M| 108B  114B|  0   0   0|   0     0 |  53    66 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M|1344B 1254B|  0   0   0|   0  10.0 | 119    85 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M| 172B  130B|  0   0   0|   0  8.00 |  80    82 | 13   7   0   0   5

Learn more about dstat on Dag Wieërs’ site.

Tagged , , , , ,

Kerberos for haters

I’ll be the first one to admit that Kerberos drives me a little insane. It’s a requirement for two of the exams in Red Hat’s RHCA certification track and I’ve been forced to learn it. It provides some pretty nice security features for large server environments. You get central single sign ons, encrypted authentication, and bidirectional validation. However, getting it configured can be a real pain due to some rather archaic commands and shells.

Here’s Kerberos in a nutshell within a two-server environment: One server is a Kerberos key distribution center (KDC) and the other is a Kerberos client. The KDC has the list of users and their passwords. Consider a situation where a user tries to ssh into the Kerberos client:

  • sshd calls to pam to authenticate the user
  • pam calls to the KDC for a ticket granting ticket (TGT) to see if the user can authenticate
  • the KDC replies to the client with a TGT encrypted with the user’s password
  • pam (on the client) tries to decrypt the TGT with the password that the user provided via ssh
  • if pam can decrypt the TGT, it knows the user is providing the right password

Now that the client has a a TGT for that user, it can ask for tickets to access other network services. What if the user who just logged in wants to access another Kerberized service in the environment?

  • client calls the KDC and asks for a ticket to grant access to the other service
  • KDC replies with two copies of the ticket:
    • one copy is encrypted with the user’s current TGT
    • a second copy is encrypted with the password of the network service the user wants to access
  • the client can decrypt the ticket which was encrypted with the current TGT since it has the TGT already
  • client makes an authenticator by taking the decrypted ticket and encrypting it with a timestamp
  • client passes the authenticator and the second copy of the ticket it received from the KDC
  • the other network service decrypts the second copy of the ticket and verifies the password
  • the other network service uses the decrypted ticket to decrypt the authenticator it received from the client
  • if the timestamp looks good, the other network service allows the user access

Okay, that’s confusing. Let’s take it one step further. Enabling pre-authentication requires that clients send a request containing a timestamp encrypted with the user’s password prior to asking for a TGT. Without this requirement, an attacker can ask for a TGT one time and then brute force the TGT offline. Pre-authentication forces the client to send a timestamped request encrypted with the user’s password back to the KDC before they can ask for a TGT. This means the attacker is forced to try different passwords when encrypting the timestamp in the hopes that they’ll get a TGT to work with eventually. One would hope that you have something configured on the KDC to set off an alarm for multiple failed pre-authentication attempts.

Oh, but we can totally kick it up another notch. What if an attacker is able to give a bad password to a client but they’re also able to impersonate the KDC? They could reply to the TGT request (as the KDC) with a TGT encrypted with whichever password they choose and get access to the client system. Enabling mutual authentication stops this attack since it forces the client to ask the KDC for the client’s own host principal password (this password is set when the client is configured to talk to the KDC). The attacker shouldn’t have any clue what that password is and the attack will be thwarted.

By this point, you’re either saying “Oh man, I don’t ever want to do this.” or “How do I set up Kerberos?”. Stay tuned if you’re in the second group. I’ll have a dead simple (or as close to dead simple as one can get with Kerberos) how-to on the blog shortly.

In the meantime, here are a few links for extra Kerberos bedtime reading:

Tagged , , , , , , , , ,