Tagged with database

Red Hat Summit 2012: Thursday

Thursday has felt like the busiest, most jam-packed day of the week. The morning started off with three keynotes from HP, Intel, and Red Hat’s CTO, Brian Stevens.

HP’s message centered around converged cloud and that customers don’t need an all or nothing solution. They can pull the best pieces from every type of hosting to do what’s best for their business. The presentation from Intel was extremely heavy on the marketing side and didn’t have much to do with Red Hat. Pauline Nist talked about how chip fabs operate, the heavy costs involved, and how their processors have changed over time. It felt a bit more like a sales pitch than anything else. She wrapped up with a pretty interesting time lapse video of the construction of a fab building and how the chips are made within the fab.

Jim Curry talking about OpenStack during the Red Hat Summit 2012 keynoteBrian Stevens talked a lot about keeping things in the open, reducing vendor lock-in, and pushing for innovation from multiple sources. He talked a lot about OpenStack and how it allows people to deliver a consistent user experience on a very open platform.

There was even a video with Rackspace’s own Jim Curry talking about the unique challenges for building a cloud orchestration layer and how OpenStack solves quite a few of those challenges.

The sessions broke out after that and I made a beeline for Dan Walsh’s session entitled “Multi-tenancy Virtualization Challenges and Solutions.” I first ran into Dan at the FUDCon in Tempe and his session there about SELinux changed my mind about it for good. He covered the basics about the dangers of multi-tenant clouds and then went through multiple examples of how to mitigate the risks. Some of the technologies mentioned included sVirt, SELinux policies, and libseccomp.

The discussion about libseccomp caught my attention because the idea is genius. We know that SELinux itself is quite solid. However, what if a user finds some other flaw in the kernel which allows them to circumvent the SELinux layer altogether? The seccomp additions to linux 3.5 allow you specify which syscalls a particular process is allowed to make. Dan gave an example of QEMU when running KVM. If QEMU only needs 20-30 syscalls to get its job done, why allow it to run every syscall available in the x86_64 and x86 instruction sets? With seccomp, you can specify which syscalls are allowed and what you want to have happen when a syscall is requested that isn’t allowed. Even if someone found a good kernel flaw, they might get blocked from doing anything malicious if the process isn’t allowed to make a syscall necessary to tear up the system.

I met up with the Beefy Miracle himself just before lunch for a photo op:

Major and the Beefy Miracle

Major and the Beefy Miracle


Another good session was entitled “Using an Open Source Framework to Catch the Bad Guy” and it covered auditd in more detail than I imagined was even possible. Mark St. Laurent talked about how auditd’s standard configuration will work for most users but that the government requires some pretty hefty adjustments. I wasn’t aware that you can actually tell auditd to halt the server if it runs out of room to write the audit log to the disk. The US Government requires this to be enabled on many machines.

The last session I attended was Sanjay Rao’s “Tuning Red Hat Systems for Databases” and it was tremendous. He talked about the differences between OLTP/DSS workloads and how all kinds of settings affect their performance. He covered everything from power management to disk elevators to NUMA. There were what seemed like a million slides and all of them contained some really good information. I really wish he could have cracked the presentation into two sessions to allow for some more discussion around details. If you can find the slides from this session, be sure to look through them.

Fenway Park Press BoxAs the day ended, a line of buses pulled up in front of the convention center and we were whisked away to Fenway Park for a private party sponsored by IBM. One of my childhood dreams was to travel to Fenway Park (and Wrigley Field) and I can say I’m halfway done! The park was amazing and it was truly an experience to walk up next to the Green Monster and look up to see how tall it really is.

The food was great and there was plenty of Sam Adams on tap. I ran into quite a few Red Hat folks and eventually found Thomas Cameron so I could thank him for the great SELinux session he ran on Wednesday. I’m going to take his slides back and share them at work to inspire some confidence around managing systems with SELinux enabled. A few twitter friends came up and we had some great conversations about cloud computing, OpenStack, and good beer. For anyone who attends class or works at Harvard, be sure to sync up with Philip Durbin. He’s a smart guy and he was a lot of fun to talk to.

Friday’s the last day and then the trip home begins. I’ll try to write up a wrap-up post tomorrow from the airport.

Tagged , , , , , , , , , ,

MySQLTuner mentioned at the O’Reilly MySQL Conference

If you push play, the video should scoot out to about the 14m40s mark where MySQLTuner appears on one of the slides. Thanks to Trent Hornibrook for letting me know!

Tagged , , , ,

Monitor MySQL restore progress with pv

The pv command is one that I really enjoy using but it’s also one that I often forget about. You can’t get a much more concise definition of what pv does than this one:

pv allows a user to see the progress of data through a pipeline, by giving information such as time elapsed, percentage completed (with progress bar), current throughput rate, total data transferred, and ETA.

The usage certainly isn’t complicated:

To use it, insert it in a pipeline between two processes, with the appropriate options. Its standard input will be passed through to its standard output and progress will be shown on standard error.

A great application of pv is when you’re restoring large amounts of data into MySQL, especially if you’re restoring data under duress due to an accidentally-dropped table or database. (Who hasn’t been there before?) The standard way of restoring data is something we’re all familiar with:

# mysql my_database < database_backup.sql

The downside of this method is that you have no idea how quickly your restore is working or when it might be done. You could always open another terminal to monitor the tables and databases as they’re created, but that can be hard to follow.

Toss in pv and that problem is solved:

# pv database_backup.sql | mysql my_database
96.8MB 0:00:17 [5.51MB/s] [==>                                ] 11% ETA 0:02:10

When it comes to MySQL, your restore rate is going to be different based on some different factors, so the ETA might not be entirely accurate.

Tagged , , , ,

A simple guide to redundant cloud hosting

Update: I’ve removed the guide as it has aged quite a bit and is not very helpful.


Today, on my 28th birthday, I’m finally delivering on a promise to my readers which I made about two months ago. I’ve written a guide on how to host a web application redundantly in a cloud environment. While it’s still a bit of a rough draft, it should be a good starting point for those who haven’t worked in virtualized environments before. Also, it may show some of the more experienced systems administrators a new way to do things.

The guide: Redundant Cloud Hosting Guide

As always, if you find anything in the guide that needs improvement, I’m all ears. :-)

Tagged , , , , , , , , , , , , , , , , , , , , , , ,

MySQL: The total number of locks exceeds the lock table size

If you’re running an operation on a large number of rows within a table that uses the InnoDB storage engine, you might see this error:

ERROR 1206 (HY000): The total number of locks exceeds the lock table size

MySQL is trying to tell you that it doesn’t have enough room to store all of the row locks that it would need to execute your query. The only way to fix it for sure is to adjust innodb_buffer_pool_size and restart MySQL. By default, this is set to only 8MB, which is too small for anyone who is using InnoDB to do anything.

If you need a temporary workaround, reduce the amount of rows you’re manipulating in one query. For example, if you need to delete a million rows from a table, try to delete the records in chunks of 50,000 or 100,000 rows. If you’re inserting many rows, try to insert portions of the data at a single time.

Further reading:

Tagged , , , , ,