Tagged with glusterfs …

Jun 28 2012
1 Comment

Red Hat Summit 2012: Wednesday

Wednesday was action-packed with dramatic keynotes and great sessions. The morning was kicked off by Paul Cormier and he talked about some new products coming from Red Hat. Much of the product releases were centered around cloud offerings (like Openshift) and his talk was mainly aimed at CIO’s and decision makers. There wasn’t a lot of technical detail within his talk but it was refreshing to hear a Linux vendor talk about their products as being revolutionary steps in pulling away from vendor lock-in and proprietary solutions.

Paul was followed by Irfan Khan who talked about the value of very low latency information exchange and processing. He drove home the point that the biggest value we can gain from information in the current age is related to our ability to gather and interpret the information in as close to real time as possible. I expected a speech from a SAP employee to be relatively dry but I was pleasantly surprised to find that he made a lot of good points. Irfan emphasized that big data providers need to find a way to fit into their customers’ landscape without causing too much disruption while also providing some real benefits.

My first session was Jeff Darcy’s discussion about Red Hat’s storage offering and what GlusterFS advancements were on the horizon. His talk was standing room only and he covered a lot of highly technical points about GlusterFS. I’m getting the feeling that GlusterFS is gaining more momentum and that we’ll be seeing more features around consistency and performance very soon.

As a fan of SELinux, I made sure that I was in Thomas Cameron’s “SELinux for Mere Mortals” class. Although I feel relatively confident that I can solve SELinux problems when I find them, Thomas covered a lot of easier solutions that I hadn’t previously considered. His explanation of the basics of SELinux are a must read for any system administrator working on a Red Hat system. I managed to find his slides from last year but he said the new slides should be up by Friday.

I attended another good class about managing network resources with Red Hat. Although the slides were a little wordy, the content was extremely good. The speaker talked about receiving a 40Gb/sec ethernet card from Mellanox and how he bumped the performance from 8Gb/sec to 37Gb/sec by adjusting CPU pinning (for NUMA) as well as some kernel configuration around TCP buffers. It was an eye-opening discussion and it was a good session for people who are trying to find bottlenecks in their hardware.

The afternoon was spend mingling with GlusterFS developers and users as well as the people working the Fedora booth. I managed to pick up some Fedora stickers but I’ve yet to get my picture taken with the life-size Beefy Miracle hot dog. That’s my goal for tomorrow.

The night wrapped up with the Red Hat Certified Professionals Party at McGreevy’s across from the Hynes Convention Center. I ran into a bunch of fellow RHCA’s and RHCE’s who read my blog and I was glad some of the posts were able to help them along their way to becoming certified. Congratulations to the folks who passed the early rounds of the new JBoss exams! Being the first ones through that process certainly can’t be easy.

For anyone who is working towards their RHCA, be sure to read my post about my experience with it. It’s a long haul, but the knowledge you’ll gain will be worth it.

Tagged , , , , , , , ,
Aug 08 2011
1 Comment

Xen Summit 2011: My Takeways

Xen Summit 2011 LogoQuite a few people who couldn’t make it to Xen Summit 2011 this year asked me to write a post summarizing my takeaways from the event. I’m not generally one to back down from peer pressure, so read on if you’re interested about the discussions at this year’s Summit.

The feeling I had at last year’s summit is that Xen was on the verge of losing traction in the market. Very few distributions still had Xen support going forward and much of the discussion was around the lack of dom0 support in upstream Linux kernels. Distribution vendors were hesitant to drag patches forward into modern kernels and this made it much more difficult to get Xen working for many people.

Major at the Golden Gate BridgeThis year was quite different. The number of attendees was up, the venue was much better, and there was an obvious buzz of energy in the room. As many of the presenters noted, this excitement stemmed from the upstream dom0 support in Linux 3.0. This inclusion is a huge win and it helps to drive Xen forward since the developers don’t have to worry about dragging patches forward. They can focus on improving performance, adding features, and tightening security.

Many of the discussions this year focused on security and performance. Ian Pratt discussed Xen’s ability to view memory pages of virtual machines via an API to detect malware running inside the instance. Memory pages could be identified and marked as not executable or applications could be triggered when a VM attempts to touch a particular memory page. Also, the whole VM could be frozen if needed.

There’s also a big push to bring code out of the dom0 and push it into utility VM’s. Driver domains could manage the network or I/O infrastructure and this would further reduce the amount of privileged code actively running in dom0. There is already very little code required for the Xen hypervisor itself (much much less than the Linux kernel — I’m looking at you, KVM) and this reduces the attack surface for potential compromises of the hypervisor. Some projects even aim to restart driver domains multiple times per minute to ensure that any malicious code injected into those virtual machines can’t exist for long periods.

Pradeep Vincent from Amazon talked about how Amazon uses Xen and the pain points they have with its current architecture. Much of his discussion was around scaling problems (and we see many of the same issues at Rackspace). Higher performance could easily be gained by multi-threaded operations in dom0 when attaching block devices and creating virtual network interfaces. He also saw some areas for performance gains in the pvops I/O code.

Quite a few of the talks centered on the ARM architecture and what Xen is able to do on those systems after Samsung published their port in 2008. HVM is on the way for ARM and it might even show up in Xen 4.2. Some demos of Xen on mobile phones from Samsung were amazing. They showed how an attacker could compromise the web browser on the phone with a keylogger, but that application was running in a VM. Once the user switched back to the phone’s main menu, the keylogger couldn’t access the keystrokes any longer. After that, a simple close of the browser killed the VM and destroyed the malicious code.

Xen 4.2 should be available in early 2012 and the feature list is staggering. Improvements to libxenlight, pvops performance (even in HVM), and guest memory sharing should be available with the new release. Nested virtualization (run a hypervisor inside a hypervisor) is also coming in Xen 4.2 and I’m sure Xzibit will be a huge fan. This should streamline hypervisor testing, allow for embedded hypervisor options and extend the capabilities of client hypervisors. Remus should be available in 4.2 as well, but it might be marked as experimental. OVMF will be added as a BIOS option for UEFI (along with the standard SeaBIOS) and this should allow for Mac OS X guests. UEFI allows Windows to boot faster since it switches to PV mode sooner and it allows for simpler platform certification for software vendors.

Mike McClurg’s presentation on XCP was pretty important to me since Rackspace is a big consumer of XenServer. If you’re not familiar with XCP, it’s basically open-source XenServer which runs on bleeding edge (and sometimes unstable) components. XCP 1.5 and XenServer 6 should be available in November with Xen 4.1 and Linux 2.6.32. GPU passthrough, up to 1TB RAM, and disaster recovery will be available. Another goal for the XCP team is to work closely with OpenStack via Project Olympus. Mike’s vision is to have XCP become the configuration of choice for open source clouds. Project Kronos was also extremely interesting. It’s essentially XCP’s XenAPI stack running on Debian and Ubuntu. You’d be able to install either OS on a physical server and run XCP’s services on it for a fully OSS hypervisor.

Konrad Wilk gave an update on Linux pvops and it appears there is a shift to get Xen working well on a desktop. This includes 3D graphics support, S3/hibernate capabilities and various bug fixes. There’s also a push to get PV functionality into HVM and get HVM functionality into PV. Driver/device domains were discussed again in Patrick Kolp’s talk and he had plenty of graphs showing performance changes when regularly restarting device domains. The performance dips were almost negligible with 10 second restarts and the security gains were significant.

There were several other great presentations on other topics like GlusterFS, OpenStack Nova, and Linpicker (from the NSA!). If these types of things interests you, keep your eyes peeled for Xen Summit 2012 next year. The weather in the bay area is well worth the trip. ;)

Tagged , , , , , , ,
Jan 30 2011
1 Comment

FUDCon 2011: Day One

The first day of FUDCon 2011 in Tempe is coming to a close tonight and I’m completely exhausted. As promised, I’ll try to summarize the day and cover the talks which I attended.

The day started out with Jared Smith’s “State of Fedora” address. The audio has already been posted on the wiki, but the speech was very positive overall. He talked about some of the struggles that have happened in the past and how they’ll probably happen again in some form or another. It was pretty inspirational and you could obviously tell that people in the room were energized by it.

After the address, all of the talks were pitched in BarCamp format. It was a very efficient and entertaining way to create a schedule for the conference. Everyone had 15-20 seconds to present their talk and then they had to rush outside to post their topic on the wall. We all had the opportunity to go outside and vote for the talks that sounded interesting. Once the votes were tallied, the schedule was set and the conference was fully underway.

The first talk for me was about Marek Goldmann’s BoxGrinder. (Note: If you Google for BoxGrinder, make sure that you enter it as a single word. You’ll get some wild unrelated results if you use two words.) In short, BoxGrinder gives you the ability to have a kickstart-ish method for automatically building images for virtual machine environments. It’s completely plugin-based, so you can have different platform and delivery plugins depending on where your VM needs to be deployed. For example, you could deploy a VM with BoxGrinder that is in a format for VMWare (platform) and is delivered to the target server via SFTP (delivery). The public cloud plugins are only compatible with Amazon’s products, but I’m eager to change that during one of the upcoming hackfests.

The Sheepdog talk started up right after lunch and although it was interesting, I think it left most people with quite a few questions when it was over. However, I think people are generally apprehensive when anyone tries to do anything innovative with storage. Losing data due to a bug is a big concern and many of the questions went deeper into data safety than performance and functionality.

Next up was Dave Malcolm’s talk about the different implementations of python. This was definitely an eye-opening talk for my coworker and I. Dave covered CPython, Jython, PyPy and various other implementations and compared their advantages and disadvantages. I’m still pretty new to Python (I’m clutching on to ruby, PHP and perl still), but this talk really had me thinking about which implementations are best for a particular environment or task. It was quite a bit of fun to learn about some of the deep underpinnings of Python and how they differ depending on the specific implementation.

Jeff Darcy’s talk about CloudFS was very intriguing. I’ve been a fan of GlusterFS recently, but I eventually moved away due to a lack of enterprise features and degrading performance. Jeff is working to add in encryption and authentication without rewriting the filesystem itself. There are quite a few tricky problems involved in the encryption portion due to partial writes and general security during the handshake process. CloudFS could potentially be a network filesystem which could be shared by multiple tenants with their own individual namespaces and segregated UID’s. This could be a big win for providers as they could offer up large amounts of storage in an organized fashion without too many management headaches.

We wrapped up the day of talks with Chris Lalancette’s presentation about Deltacloud. In short, it’s a bag of daemons that allow you to manage multiple public or private clouds. Everything from image management to provisioning are included in the project. Questions were raised about whether another application was needed since vendor-specific libraries are abundant and libcloud offers many of the same features in a simpler package.

Tonight’s social event was FUDPub at ASU’s Memoral Union building. The food and drinks were excellent (thanks to Rackspace!) and it was a great opportunity to relax and talk with other Fedora users and developers. We had the opportunity to meet people from around the world while playing round after round of bowling and billiards. The discussions were extremely valuable, but as I said before, it was quite tiring.

I’ve compiled the FUDCon photos I’ve taken into a Flickr photo set.

That’s the end of today’s summary. I’ll try to keep this going tomorrow as well. Thanks for reading!

Tagged , , , , , , , ,
Nov 10 2010
5 Comments

Switching from GlusterFS to DRBD and OCFS2

As my uptime reports have shown, and as some of you have reported, my blog’s load time has increased steadily over the past few weeks. It turns out that one of my VM’s was on a physical machine that had some trouble and I was reaching a point where GlusterFS’s replicate functionality couldn’t meet my performance needs.

Instead of using GlusterFS as I had before in my redundant cloud hosting guide, I decided to use DRBD in dual-primary mode with OCFS2 as the clustering filesystem on top of it. The performance is quite good so far:

Pingdom Response Time Graph for rackerhacker.com

Pingdom Response Time Graph for rackerhacker.com

I switched over the DNS late last night and the response time has fallen from the two to three second range (during times of low load) to right around one second per request. In addition to the reduced load times, I can support higher concurrency without significant performance degradation.

Don’t worry – I’ll make a detailed post on this topic later along with a guide on how to set it up yourself.

Tagged , , , , , ,
Aug 13 2010
2 Comments

Very unscientific GlusterFS benchmarks

I’ve been getting requests for GlusterFS benchmarks from every direction lately and I’ve been a bit slow on getting them done. You may suspect that you know the cause of the delays, and you’re probably correct. ;-)

Quite a few different sites argue that the default GlusterFS performance translator configuration from glusterfs-volgen doesn’t allow for good performance. You can find other sites which say you should stick with the defaults that come from the script. I decided to run some simple tests to see which was true in my environment.

Here’s the testbed:

  • GlusterFS 3.0.5 running on RHEL 5.4 Xen guests with ext3 filesystems
  • one GlusterFS client and two GlusterFS servers are running in separate Xen guests
  • cluster/replicate translator is being used to keep the servers in sync
  • the instances are served by a gigabit network

It’s about time for some pretty graphs, isn’t it?

iozone re-reader benchmark results with default glusterfs translators from glusterfs-volgeniozone re-reader benchmark results with no glusterfs translators

The test run on the left used default stock client and server volume files as they come from glusterfs-volgen. The test run on the right used a client volume file with no performance translators (the server volume file was untouched). Between each test run, the GlusterFS mount was unmounted and remounted. I repeated this process four times (for a total of five runs) and averaged the data.

You’ll have to forgive the color mismatches and the lack of labeling on the legend (that’s KB/sec transferred) as I’m far from an Excel expert.

The graphs show that running without any translators at all will drastically hinder read caching in GlusterFS — exactly as I expected. Without any translators, the performance is very even across the board. Since my instances had 256MB of RAM each, their iocache translator was limited to about 51MB of cache. That’s reflected in the graph on the left — look for the vertical red/blue divider between the 32MB and 64MB file sizes. I’ll be playing around with that value soon to see how it can improve performance for large and small files.

Keep in mind that this test was very unscientific and your results may vary depending on your configuration. While I hope to have more detailed benchmarks soon, this should help some of the folks who have been asking for something basic and easy to understand.

Tagged , , ,
← Older posts