Mysql on Major Hayden

Fixing a ghost database migration failure

major@mhtx.net (Major Hayden) — Thu, 31 Aug 2023 00:00:00 +0000

I love learning about the behind the scenes aspects of just about everything. I do ham radio, I self-host lots of my personal infrastructure, and I’ve been learning more about the math behind the stock market for the last year or two.

That led me to start a blog on Ghost to share my findings with others. I started Theta Nerd¹ earlier this summer.

My deployment looked great when I started! Everything was automatically updated with watchtower and running with docker-compose on Fedora CoreOS. (Click these links to read the posts on both topics!)

However, I woke up one morning to my monitoring going off and my site was down. 😱

Why is the site down?

Anyone who has worked in IT knows this sinking feeling. Something is down, you don’t know why, and you suspect the worst possible scenarios.

The instance hosting the blog was online and responsive, so I started digging into the logs with docker-compose logs. I suddenly found a wall of text in the logs for the Ghost container:

[2023-08-03 11:10:16] INFO Adding members.email_disabled column
[2023-08-03 11:10:16] INFO Setting email_disabled to true for all members that have their email on the suppression list
[2023-08-03 11:10:16] INFO Setting nullable: stripe_products.product_id
[2023-08-03 11:10:16] INFO Adding table: donation_payment_events
[2023-08-03 11:10:16] INFO Rolling back: alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column 'member_id' and referenced column 'id' in foreign key constraint 'donation_payment_events_member_id_foreign' are incompatible..
[2023-08-03 11:10:16] INFO Dropping table: donation_payment_events
[2023-08-03 11:10:16] INFO Dropping nullable: stripe_products.product_id with foreign keys disabled
[2023-08-03 11:10:16] INFO Setting email_disabled to false for all members
[2023-08-03 11:10:16] INFO Removing members.email_disabled column
[2023-08-03 11:10:16] INFO Rollback was successful.
[2023-08-03 11:10:16] ERROR alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column 'member_id' and referenced column 'id' in foreign key constraint 'donation_payment_events_member_id_foreign' are incompatible.

alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column 'member_id' and referenced column 'id' in foreign key constraint 'donation_payment_events_member_id_foreign' are incompatible.
{"config":{"transaction":false},"name":"2023-07-27-11-47-49-create-donation-events.js"}
"Error occurred while executing the following migration: 2023-07-27-11-47-49-create-donation-events.js"
Error ID:
 300
Error Code: 
 ER_FK_INCOMPATIBLE_COLUMNS
----------------------------------------
Error: alter table `donation_payment_events` add constraint `donation_payment_events_member_id_foreign` foreign key (`member_id`) references `members` (`id`) on delete SET NULL - Referencing column 'member_id' and referenced column 'id' in foreign key constraint 'donation_payment_events_member_id_foreign' are incompatible.
 at /var/lib/ghost/versions/5.57.2/node_modules/knex-migrator/lib/index.js:1032:19
 at Packet.asError (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/packets/packet.js:728:17)
 at Query.execute (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/commands/command.js:29:26)
 at Connection.handlePacket (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/connection.js:478:34)
 at PacketParser.onPacket (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/connection.js:97:12)
 at PacketParser.executeStart (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/packet_parser.js:75:16)
 at Socket.<anonymous> (/var/lib/ghost/versions/5.57.2/node_modules/mysql2/lib/connection.js:104:25)
 at Socket.emit (node:events:513:28)
 at addChunk (node:internal/streams/readable:315:12)
 at readableAddChunk (node:internal/streams/readable:289:9)
 at Socket.Readable.push (node:internal/streams/readable:228:10)
 at TCP.onStreamRead (node:internal/stream_base_commons:190:23)

Ah, so a failed database migration in the upgrade to 5.57.2 is the culprit! 👏

I brought the site back online quickly by changing the container version for Ghost back to the previous version (5.55.2).

Why did the database migration fail?

The error message from above boils down to this:

Error: alter table `donation_payment_events` add constraint 
`donation_payment_events_member_id_foreign` foreign key (`member_id`)
references `members` (`id`) on delete SET NULL - Referencing column
'member_id' and referenced column 'id' in foreign key constraint 
'donation_payment_events_member_id_foreign' are incompatible.

Adjusting the donation_payment_events.member_id column to be a foreign key of members.id is failing because they are incompatible types. However, as I examined both tables, both were regular varchar(24) columns without anything special attached to them:

mysql> describe members;
+------------------------------+---------------+------+-----+---------+-------+
| Field | Type | Null | Key | Default | Extra |
+------------------------------+---------------+------+-----+---------+-------+
| id | varchar(24) | NO | PRI | NULL | |
| uuid | varchar(36) | YES | UNI | NULL | |
| email | varchar(191) | NO | UNI | NULL | |
| status | varchar(50) | NO | | free | |
| name | varchar(191) | YES | | NULL | |
| expertise | varchar(191) | YES | | NULL | |
| note | varchar(2000) | YES | | NULL | |
| geolocation | varchar(2000) | YES | | NULL | |
| enable_comment_notifications | tinyint(1) | NO | | 1 | |
| email_count | int unsigned | NO | | 0 | |
| email_opened_count | int unsigned | NO | | 0 | |
| email_open_rate | int unsigned | YES | MUL | NULL | |
| last_seen_at | datetime | YES | | NULL | |
| last_commented_at | datetime | YES | | NULL | |
| created_at | datetime | NO | | NULL | |
| created_by | varchar(24) | NO | | NULL | |
| updated_at | datetime | YES | | NULL | |
| updated_by | varchar(24) | YES | | NULL | |
+------------------------------+---------------+------+-----+---------+-------+
18 rows in set (0.00 sec)

Going upstream

I went to Ghost’s GitHub repository and opened an issue with as much data as I can find.

One of the first replies mentioned something about database collations. Long story short, collations describe how databases handle sorting and comparing data for different languages. Comparing some languages to other languages can be particularly challenging and this can lead to problems.

I made a switch from MariaDB to MySQL recently for the blog. Could that be related?

More searching

I figured that I wasn’t the first one to stumble into this problem, and sure enough – I wasn’t! There’s a great blog post about a broken migration from MySQL 5 to 8 with Ghost.

In short, it required several steps to fix it:

Stop the Ghost container
Back up the database first (always a good idea)
Do a quick find/replace on the dumped database to change the collations
Drop the ghost database from the database 😱
Import the database back into MySQL
Start Ghost again

Dropping databases always makes me pause, but that’s what backups are for! 😉

How I fixed it

In my case, my MySQL container is called ghostmysql and my Ghost database is ghostdb. Then I made a backup of the database using mysqldump:

sudo docker-compose exec ghostmysql mysqldump \
 -u root -psuper-secret-password ghostdb > backup-ghost-db.sql

Next, I copied the SQL file to another directory just in case I accidentally deleted this backup with an errant command.

cp backup-ghost-db.sql ../

Then I made a copy of the SQL file in the current directory and ran the find and replace on that copy. This changes the collations from the wrong one, utf8mb4_general_ci, to the right one, utf8mb4_0900_ai_ci²:

cp backup-ghost-db.sql backup-ghost-db-new.sql
sed -i 's/utf8mb4_general_ci/utf8mb4_0900_ai_ci/g' \
 backup-ghost-db-new.sql

Now I have the collations right for importing the database back into MySQL. But first, I have to drop the existing database. This is a good time to double check your backups!

sudo docker-compose exec ghostmysql mysql -u root \
 -psuper-secret-password
mysql> DROP DATABASE ghostdb;

Now we can import the modified backup:

cat backup-ghost-db-new.sql | sudo docker-compose exec -T \
 ghostmysql mysql -u root -psuper-secret-password ghostdb

Start all the containers:

sudo docker-compose up -d

Ghost was back online with the older version and everything looked good! I updated my docker-compose.yaml back to use latest for the Ghost version and ran sudo docker-compose up -d once more.

Within seconds, the new container image was in place and the container was running! Both migrations completed in seconds and the blog was back online with the newest version. 🎉

Theta is one of many financial Greeks that measure certain aspects of options contracts in the market. It’s also a letter in the Greek alphabet. ↩︎
The default collation in MySQL 8 is utf8mb4_0900_ai_ci. ↩︎

Survive the Google Reader exodus with Tiny Tiny RSS

major@mhtx.net (Major Hayden) — Sun, 17 Mar 2013 21:27:38 +0000

It’s no secret that Google Reader is a popular way to keep up with your RSS feeds, but it’s getting shelved later this year. Most folks suggested Feedly as a replacement but I found the UI quite clunky in a browser and on Android devices.

Then someone suggested Tiny Tiny RSS. I couldn’t learn more about it on the day Google Reader’s shutdown was announced because the site was slammed. In a nutshell, Tiny Tiny RSS is a well-written web UI for managing feeds and a handy API for using it with mobile applications. The backend code is written in PHP and it supports MySQL and Postgres.

There’s also an Android application that gives you a seven day trial once you install it. The pro key costs $1.99.

The installation took me a few minutes and then I was off to the races. I’d recommend implementing SSL for accessing your installation (unless you like passing credentials in the clear) and enable keepalive connections in Apache. The UI in the application drags down a ton of javascript as it works and enabling keepalives will keep your page load times low.

If you want to get your Google Reader feeds moved over in bulk, just export them from Google Reader:

Click the settings cog at the top right of Google Reader and choose Reader Settings
Choose Import/Export from the menu
Press Export, head over to Google Takeout and download your zip file

Unzip the file and find the .xml file. Open up a browser, access Tiny Tiny RSS and do this:

Click Actions > Preferences
Click the Feeds tab
Click the OPML button at the bottom
Import the xml file that was in the zip file from Google

From there, just choose a method for updating feeds and you should be all set!

mysql-json-bridge: a simple JSON API for MySQL

major@mhtx.net (Major Hayden) — Thu, 29 Mar 2012 02:34:53 +0000

My quest to get better at Python led me to create a new project on GitHub. It’s called mysql-json-bridge and it’s ready for you to use.

Why do we need a JSON API for MySQL?

The real need sprang from a situation I was facing daily at Rackspace. We have a lot of production and pre-production environments which are in flux but we need a way to query data from various MySQL servers for multiple purposes. Some folks need data in ruby or python scripts while others need to drag in data with .NET and Java. Wrestling with the various adapters and all of the user privileges on disparate database servers behind different firewalls on different networks was less than enjoyable.

That’s where this bridge comes in.

The bridge essentially gives anyone the ability to talk to multiple database servers across different environments by talking to a single endpoint with easily configurable security and encryption. As long as the remote user can make an HTTP POST and parse some JSON, they can query data from multiple MySQL endpoints.

How does it work?

It all starts with a simple HTTP POST. I’ve become a big fan of the Python requests module. If you’re using it, this is all you need to submit a query:

import requests
payload = {'sql': 'SELECT * FROM some_tables WHERE some_column=some_value'}
url = "http://localhost:5000/my_environment/my_database"
r = requests.post(url, data=payload)
print r.text

The bridge takes your query and feeds it into the corresponding MySQL server. When the results come back, they’re converted to JSON and returned via the same HTTP connection.

What technology does it use?

Flask does the heavy lifting for the HTTP requests and Facebook’s Tornado database class wraps the MySQLdb module in something a little more user friendly. Other than those modules, PyYAML and requests are the only other modules not provided by the standard Python libraries.

Is it fast?

Yes. I haven’t done any detailed benchmarks on it yet, but the overhead is quite low even with a lot of concurrency. The biggest slowdowns come from network latency between you and the bridge or between the bridge and the database server. Keep in mind that gigantic result sets will take a longer time to transfer across the network and get transformed into JSON.

I found a bug. I have an idea for an improvement. You’re terrible at Python.

All feedback (and every pull request) is welcome. I’m still getting the hang of Python (hey, I’ve only been writing in it seriously for a few weeks!) and I’m always eager to learn a new or better way to accomplish something. Feel free to create an issue in GitHub or submit a pull request with a patch.

Monitor MySQL restore progress with pv

major@mhtx.net (Major Hayden) — Wed, 24 Nov 2010 16:43:28 +0000

The pv command is one that I really enjoy using but it’s also one that I often forget about. You can’t get a much more concise definition of what pv does than this one:

pv allows a user to see the progress of data through a pipeline, by giving information such as time elapsed, percentage completed (with progress bar), current throughput rate, total data transferred, and ETA.

The usage certainly isn’t complicated:

To use it, insert it in a pipeline between two processes, with the appropriate options. Its standard input will be passed through to its standard output and progress will be shown on standard error.

A great application of pv is when you’re restoring large amounts of data into MySQL, especially if you’re restoring data under duress due to an accidentally-dropped table or database. (Who hasn’t been there before?) The standard way of restoring data is something we’re all familiar with:

# mysql my_database < database_backup.sql

The downside of this method is that you have no idea how quickly your restore is working or when it might be done. You could always open another terminal to monitor the tables and databases as they’re created, but that can be hard to follow.

Toss in pv and that problem is solved:

# pv database_backup.sql | mysql my_database
96.8MB 0:00:17 [5.51MB/s] [==> ] 11% ETA 0:02:10

When it comes to MySQL, your restore rate is going to be different based on some different factors, so the ETA might not be entirely accurate.

Private network interfaces: the forgotten security hole

major@mhtx.net (Major Hayden) — Tue, 02 Mar 2010 00:55:07 +0000

Regardless of the type of hosting you’re using - dedicated or cloud - it’s important to take network interface security seriously. Most often, threats from the internet are the only ones mentioned. However, if you share a private network with other customers, you have just as much risk on that interface.

Many cloud providers allow you access to a private network environment where you can exchange data with other instances or other services offered by the provider. The convenience of this access comes with a price: other instances can access your instance on the private network just as easily as they could on the public interface.

Here are some security tips for your private interfaces:

Disable the private interface

This one is pretty simple. If you have only one instance or server, and you don’t need to communicate privately with any other instances, just disable the interface. Remember to configure your networking scripts to leave the interface disabled after reboots.

Use packet filtering

The actual mechanism will vary based on your operating system, but filtering packets is the one of the simplest ways to secure your private interface. You can take some different approaches with them, but I find the easiest method is to allow access from your other instances and reject all other traffic.

For additional security, you can limit access based on ports as well as source IP addresses. This could prevent an attacker from having easy access to your other instances if they’re able to break into one of them.

Configure your daemons to listen on the appropriate interfaces

If there are services that don’t need to be listening on the private network, don’t allow them to listen on your private interface. For example, MySQL might need to listen on the private interface so the web server can talk to it, but apache won’t need to listen on the private interface. This reduces the profile of your instance on the private network and makes it a less likely target for attack.

Use hosts.allow and hosts.deny

Many new systems administrators forget about how handy tcpwrappers can be for limiting access. If your firewall is down in error, host.allow and hosts.deny could be an extra layer of protection. It’s important to ensure that the daemons you are attempting to control are build with tcpwrappers support. Daemons like sshd support it, but apache and MySQL do not.

Encrypt all traffic on the private network

Just because it’s called a “private” network doesn’t mean that your traffic can traverse the network privately. You should always err on the side of caution and encrypt all traffic traversing the private network. You can use ssh tunnels, stunnel, or the built-in SSL features found in most daemons.

This also brings up an important point: you should know how your provider’s private network works. Are there safeguards to prevent sniffing? Could someone else possibly ARP spoof your instance’s private IP addresses? Is your private network’s subnet shared among many customers?

With all of that said, it’s also very important to have proper change control policies so that administrators working after you are fully aware of the security measures in place and why they are important. This will ensure that all of the administrators on your instances will understand the security of the system and they should be able to make sensible adjustments later for future functionality.

MySQL: The total number of locks exceeds the lock table size

major@mhtx.net (Major Hayden) — Tue, 16 Feb 2010 18:00:29 +0000

If you’re running an operation on a large number of rows within a table that uses the InnoDB storage engine, you might see this error:

ERROR 1206 (HY000): The total number of locks exceeds the lock table size

MySQL is trying to tell you that it doesn’t have enough room to store all of the row locks that it would need to execute your query. The only way to fix it for sure is to adjust innodb_buffer_pool_size and restart MySQL. By default, this is set to only 8MB, which is too small for anyone who is using InnoDB to do anything.

If you need a temporary workaround, reduce the amount of rows you’re manipulating in one query. For example, if you need to delete a million rows from a table, try to delete the records in chunks of 50,000 or 100,000 rows. If you’re inserting many rows, try to insert portions of the data at a single time.

MySQL: The total number of locks exceeds the lock table size

major@mhtx.net (Major Hayden) — Fri, 29 Jan 2010 13:12:21 +0000

This problem has cropped up for me a few times, but I’ve always forgotten to make a post about it. If you’re working with a large InnoDB table and you’re updating, inserting, or deleting a large volume of rows, you may stumble upon this error:

ERROR 1206 (HY000): The total number of locks exceeds the lock table size

InnoDB stores its lock tables in the main buffer pool. This means that the number of locks you can have at the same time is limited by the innodb_buffer_pool_size variable that was set when MySQL was started. By default, MySQL leaves this at 8MB, which is pretty useless if you’re doing anything with InnoDB on your server.

Luckily, the fix for this issue is very easy: adjust innodb_buffer_pool_size to a more reasonable value. However, that fix does require a restart of the MySQL daemon. There’s simply no way to adjust this variable on the fly (with the current stable MySQL versions as of this post’s writing).

Before you adjust the variable, make sure that your server can handle the additional memory usage. The innodb_buffer_pool_size variable is a server wide variable, not a per-thread variable, so it’s shared between all of the connections to the MySQL server (like the query cache). If you set it to something like 1GB, MySQL won’t use all of that up front. As MySQL finds more things to put in the buffer, the memory usage will gradually increase until it reaches 1GB. At that point, the oldest and least used data begins to get pruned when new data needs to be present.

So, you need a workaround without a MySQL restart?

If you’re in a pinch, and you need a workaround, break up your statements into chunks. If you need to delete a million rows, try deleting 5-10% of those rows per transaction. This may allow you to sneak under the lock table size limitations and clear out some data without restarting MySQL.

To learn more about InnoDB’s parameters, visit the MySQL documentation.

PHPMyAdmin 3.x hides the table indexes

major@mhtx.net (Major Hayden) — Sat, 04 Apr 2009 00:51:48 +0000

Users of PHPMyAdmin 3.x may find that the table indexes are automatically hidden at the bottom of the page. I find this to be a huge annoyance since table indexes are tremendously important to the structure of the table.

If you don’t want to downgrade to PHPMyAdmin 2.x, just add the following line to the top of your config.inc.php file:

$cfg['InitialSlidersState'] = 'open';

This will cause the indexes to be displayed when you click Structure for a certain table. By default, they are hidden.

Sidenote: Some of you might be thinking: “Hey, you’re a DBA, you should know MySQL queries without needing PHPMyAdmin.” You’re right. I do know how to get the job done without PHPMyAdmin, but I enjoy the way PHPMyAdmin allows me to visualize my table structures. Also, it’s a handy way to present data to others very quickly.

Importing Excel files into MySQL with PHP

major@mhtx.net (Major Hayden) — Fri, 07 Nov 2008 19:42:45 +0000

If you have Excel files that need to be imported into MySQL, you can import them easily with PHP. First, you will need to download some prerequisites:

PHPExcelReader - http://sourceforge.net/projects/phpexcelreader/

Spreadsheet_Excel_Writer - http://pear.php.net/package/Spreadsheet_Excel_Writer

Once you’ve downloaded both items, upload them to your server. Your directory listing on your server should have two directories: Excel (from PHPExcelReader) and Spreadsheet_Excel_Writer-x.x.x (from Spreadsheet_Excel_Writer). To work around a bug in PHPExcelReader, copy oleread.inc from the Excel directory into a new path:

Spreadsheet/Excel/Reader/OLERead.php

The PHPExcelReader code will expect OLERead.php to be in that specific location. Once that is complete, you’re ready to use the PHPExcelReader class. I made an example Excel spreadsheet like this:

Name Extension Email
----------------------------------------------------
Jon Smith 2001 jsmith@domain.com
Clint Jones 2002 cjones@domain.com
Frank Peterson 2003 fpeterson@domain.com

After that, I created a PHP script to pick up the data and insert it into the database, row by row:

require_once 'Excel/reader.php';
$data = new Spreadsheet_Excel_Reader();
$data->setOutputEncoding('CP1251');
$data->read('exceltestsheet.xls');

$conn = mysql_connect("hostname","username","password");
mysql_select_db("database",$conn);

for ($x = 2; $x < = count($data->sheets[0]["cells"]); $x++) {
 $name = $data->sheets[0]["cells"][$x][1];
 $extension = $data->sheets[0]["cells"][$x][2];
 $email = $data->sheets[0]["cells"][$x][3];
 $sql = "INSERT INTO mytable (name,extension,email)
 VALUES ('$name',$extension,'$email')";
 echo $sql."\n";
 mysql_query($sql);
}

After the script ran, each row had been added to the database table successfully. If you have additional columns to insert, just repeat these lines, using an appropriate variable for each column:

sheets[0]["cells"][$row_number][$column_number];

For more details, you can refer to a post in Zend’s Developer Zone.

MySQL: ERROR 1040: Too many connections

major@mhtx.net (Major Hayden) — Tue, 24 Jun 2008 17:00:47 +0000

If you run a fairly busy and/or badly configured MySQL server, you may receive something like this when attempting to connect:

# mysql
ERROR 1040: Too many connections

MySQL is telling you that it is handling the maximum connections that you have configured it to handle. By default, MySQL will handle 100 connections simultaneously. This is very similar to the situation when Apache reaches the MaxClients setting. You won’t even be able to connect to MySQL to find out what is causing the connections to be used up, so you will be forced to restart the MySQL daemon to troubleshoot the issue.

What causes MySQL to run out of connections? Here’s a list of reasons that may cause MySQL to run out of available connections, listed in order of what you should check:

Bad MySQL configuration

Verify that you have set MySQL’s buffers and caches to appropriate levels for the type of data you’re storing and the types of queries that you are running. One quick way to check this information is via MySQLTuner. The script will tell you how well your server is performing along with the corrections you should make. Running the script only takes a few moments and it doesn’t require a DBA to decipher the results.

Data storage techniques

Remember that MySQL works best when moving vertically, not horizontally. If you have a table with 20 columns, breaking it into two tables with 10 columns each will improve performance. Even if you need to join the two tables together to get your data, it will still perform at a higher level. Also, use the right data types for the right data. If you’re storing an integer only, don’t use a CHAR or VARCHAR data type. If your integer will be small, then use something like a TINYINT or SMALLINT rather than INT. This means MySQL will use less memory, pull less data from the disk, and have higher performing joins.

Slow queries

These are generally pretty easy to fix. If you have queries that don’t use indexes, or if queries run slowly with indexes in place, you need to rethink how you’re pulling your data. Should your data be split into multiple tables? Are you pulling more data than you need? Keep these questions in mind, enable the slow query log, and re-work your queries to find where the bottlenecks occur.

Division of labor

Most people who use MySQL have a dynamic site written in a scripting language, like PHP, Perl or Python. It’s obvious that your server will need to do some work to parse the scripts, send data back to the client, and communicate with MySQL. If you find that your server is overworked, consider moving MySQL to its own dedicated hardware. Among many other things, this will reduce your disk I/O, allow you to better utilize memory, and it will help you when you need to scale even further. Be sure to keep your MySQL server close to your web servers, however, as increased latency will only make your performance problem first.

Right hardware

Do you have the right hardware for the job? Depending on your budget, you may need to make the move for hardware that gives you better I/O throughput and more useable cores. MySQL is a multi-threaded application, so it can utilize multiple cores to serve data quickly. Also, writing logs, reading tables, and adjusting indexes are disk-intensive tasks that need fast drives to perform well. When you look for a dedicated server for MySQL, be sure to choose multiple-core machines with low latency RAM, fast drives (SCSI/SAS), and a reliable network interface.

By reviewing these bottlenecks, you can reduce the load on your MySQL server without increasing your maximum connections. Simply increasing the maximum connections is a very bad idea. This can cause MySQL to consume unnecessary resources on your server and it may lead to an unstable system (crash!).

Remove backticks from MySQL dumps

major@mhtx.net (Major Hayden) — Wed, 18 Jun 2008 17:00:01 +0000

I found myself in a peculiar situation last week. I’d been asked to downgrade a server from MySQL 4.1 to MySQL 3.23. Believe me, I tried to advise against the request, but I didn’t succeed.

I made a MySQL 3.23 compatible dump with --compatible=mysql323, but the dump came out with backticks around the database names. This works with some 3.23 versions, but it doesn’t work with others. Apparently RHEL 3’s MySQL 3.23 is one of those versions where it simply won’t work.

This sed line came in handy to strip the backticks from the USE lines in the dump:

sed -e "s/^USE \`\(.*\)\`/USE \1/g"

MySQL: Can’t drop one or more of the requested users

major@mhtx.net (Major Hayden) — Wed, 11 Jun 2008 23:59:37 +0000

MySQL has quite a few cryptic error messages, and this one is one of the best:

mysql> DROP USER 'forums'@'db1.myserver.com';
ERROR 1268 (HY000): Can't drop one or more of the requested users

Naturally, I was quite interested to know why MySQL wasn’t going to allow me to remove this user. There was nothing special about the user, but then again, this wasn’t a server that I personally managed, so I wasn’t sure what kind of configuration was in place.

It’s always a good idea to get your bearings, so I checked the current grants:

mysql> SHOW GRANTS FOR 'forums'@'db1.myserver.com';
+----------------------------------------------------------------------+
| Grants for forums@db1.myserver.com |
+----------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'forums'@'db1.myserver.com' WITH GRANT OPTION |
+----------------------------------------------------------------------+
1 row in set (0.00 sec)

The GRANT OPTION was causing my grief. It was the only privilege that the user had on the server. I revoked the privilege and attempted to drop the user yet again:

mysql> REVOKE GRANT OPTION ON *.* FROM 'forums'@'db1.myserver.com';
Query OK, 0 rows affected (0.00 sec)
mysql> DROP USER 'forums'@'db1.myserver.com';
Query OK, 0 rows affected (0.00 sec)

It’s key to remember that revoking the GRANT OPTION is a completely separate process. Revoking ALL PRIVILEGES doesn’t include GRANT OPTION, so be sure to specify it separately:

mysql> REVOKE ALL PRIVILEGES, GRANT OPTION ON *.* FROM 'user'@'host';

Backing up MySQL to Amazon’s S3

major@mhtx.net (Major Hayden) — Fri, 06 Jun 2008 00:18:49 +0000

I received an e-mail from Tim Linden about a post he made in his blog about backing up MySQL data to Amazon’s S3.

The article goes over installing the Net::Amazon::S3 Perl module via WHM (which is handy for the cPanel users). However, if you’re not a cPanel user, you can install it via CPAN:

# perl -MCPAN -e 'install Net::Amazon::S3'

If you’d rather install it through Webmin, go to the ‘Others’ section, and click ‘Perl Modules’.

Also, Tim mentions configuring a Firefox extension for accessing S3 that works very well. However, I find myself using Safari most often, so I prefer to use Jungle Disk or Transmit on my Mac.

Overall, it’s a great post, and I’m glad Tim told me about it!

Importing MySQL dumps made on 64-bit servers

major@mhtx.net (Major Hayden) — Fri, 21 Mar 2008 17:51:56 +0000

It’s tough to find examples of dumps that can’t be properly reimported on other servers. However, if you have a 64-bit server, and you make a MySQL dump file from it, you may see this issue when importing the dump on a 32-bit MySQL server:

ERROR 1118 (42000) at line 1686: Row size too large. The maximum row size for the used table type, not counting BLOBs, is 65535. You have to change some columns to TEXT or BLOBs

You really don’t have any options in this situation. You’ll need to adjust your table on the 64-bit server for good and then make a new dump file, or you will just have to live with the fact that it can’t be imported into a 32-bit instance of MySQL.

Plesk and MySQL 5

major@mhtx.net (Major Hayden) — Wed, 30 Jan 2008 18:29:18 +0000

One of the questions I receive the most is: “What version of Plesk works with MySQL 5?” The minimum version of Plesk for MySQL 5 is 8.1.0. If you install MySQL 5 on a version prior to 8.1.0, you may be able to access then panel in the other 8.x versions, but your upgrades will fail miserably.

In case you’re curious about a slightly older system, full MySQL 4 support was available in Plesk 7.5.3. However, MySQL 4 is supported on some distributions as far back as 7.1:

Fedora Core 2

Mandrake 10

SuSE 9.0

FreeBSD 5.2.1

Check out SWSoft/Parallel’s site for more information about MySQL 4 and 5 support.

MySQL Replication: Wrap-up

major@mhtx.net (Major Hayden) — Tue, 15 Jan 2008 18:20:02 +0000

After a couple of weeks, my MySQL replication series has come to a close. Here’s links to all of the topics that I covered:

Performance

Redundancy

Backups and Data Integrity

Horizontal Data Partitioning

Delayed Slaves

Breakdown

Replication Across an External Network

Upgrading the MySQL Server

Slave Performance

If there’s any other questions that you have, please let me know and I’ll be happy to add some extra posts on your topic!

MySQL Replication: Slave Performance

major@mhtx.net (Major Hayden) — Mon, 14 Jan 2008 18:26:40 +0000

There’s a few final configuration options that may help the performance of your slave MySQL servers. If you’re not using certain storage engines, like InnoDB or Berkeley, then by all means, remove them from your configuration. For those two specifically, just add the following to your my.cnf on the slave server:

skip-innodb<br /> skip-bdb

To reduce disk I/O on big MyISAM write operations, you can delay the flushing of indexes to the disk:

delay_key_write = ALL

You can also make all of your write queries take a backseat to any reads:

low-priority-updates

Keep in mind, however, that the last two options will increase slave performance, but it may cause them to lag behind the master. Depending on your application, this may not be acceptable.

MySQL Replication: Upgrading the MySQL server

major@mhtx.net (Major Hayden) — Fri, 11 Jan 2008 23:44:54 +0000

If you want to make a DBA nervous, just let them know that they need to upgrade MySQL servers that are replicating in a production environment. There’s multiple ways to get the job done, but here is the safest route:

First, make sure you have dumped all of your databases properly. Verify that your backups are correct and intact, and that you have multiple copies of them.

Next, upgrade the slave servers individually to the newest version. After upgrading the first one, make sure the slave server is operating properly. If it is working properly, then you can continue to upgrade the other slaves.

Once all of the slaves have been upgraded, then you can upgrade the master. If a busy web application is sending write queries to the master, you may want to put up a temporary page that tells visitors that maintenance is being performed. Once all of the writes clear out, stop the master and upgrade it.

After the master starts up, be sure that the slaves reconnect, and you might want to perform a test write query. Verify that the write is performed on the slaves as it was done on the master.

MySQL Replication: Across an external network

major@mhtx.net (Major Hayden) — Thu, 10 Jan 2008 18:51:39 +0000

While many people might find replicating over an external network to be an odd concept, it does have some uses. For example, if you need to replicate data for local access at certain locations, it may be helpful. Also, if you have a dedicated server, you can replicate to your home to run backups.

First off, you’re going to need security for the connection. This is easily done with SSL. On the master, simply add the following lines to the [mysqld] section and restart the master:

ssl-ca=cacert.pem<br /> ssl-cert=server-cert.pem<br /> ssl-key=server-key.pem

To have the slaves use SSL connections to the master server, simply add on MASTER_SSL=1 to the CHANGE MASTER statement on the slave.

Another aspect to consider is bandwidth usage. This may be a priority if your remote areas have slow downlinks, or if you are charged for your bandwidth usage. You can compress the MySQL traffic very easily. Simply add the following to the MySQL configuration file in the [mysqld] section:

slave_compressed_protocol = 1

With both of these changes, keep in mind that there is a significant CPU overhead required to compress and/or encrypt data. Determine carefully what your application requires and test your configuration thoroughly.

MySQL Replication: Breakdown

major@mhtx.net (Major Hayden) — Wed, 09 Jan 2008 18:24:03 +0000

On some occasions, MySQL replication can break down if an statement comes from the master that makes no sense to the slave. For example, if an UPDATE statement arrives from the master server, but the table referenced by the UPDATE no longer exists, then the slave will halt replication and throw an error when SHOW SLAVE STATUS; is run.

The obvious question here is: how can the master and the slave have different data after replication has started? After all, you make a dump file prior to starting replication, so both servers contain the same information. Stray updates can be thrown into the mix from application errors or plain user errors. These kinds of things happen, even though we all try to avoid it.

Don’t worry - this is almost always an easy fix. You have two main options:

Fix the problem yourself. If the master sent a query that the slave can’t run, fix it manually. For example, if the master wants to run an INSERT on a table that doesn’t exist, run a quick SHOW CREATE TABLE; on the master and create the table manually on the slave. When the table is there, run a START SLAVE; on the slave and you should be all set.

Skip an unnecessary query. Let’s say that the master sent over a DROP TABLE query but the table doesn’t exist on the master. It’s safe to say that the master won’t be sending any write queries to that table in the future, so the query can be skipped. To skip it, run the following statement:

mysql> SET GLOBAL SQL_SLAVE_SKIP_COUNTER = 1;<br /> mysql> START SLAVE;

In short, you’re telling MySQL to skip that unnecessary query and keep going with the ones after that. Of course, if you need to skip multiple queries, change the 1 to whatever number you need and then run START SLAVE;.

MySQL Replication: Delayed Slaves

major@mhtx.net (Major Hayden) — Wed, 09 Jan 2008 03:11:11 +0000

In a perfect world, slaves will contain the same data as the master at all times. The events should be picked up and executed by the slaves in milliseconds. However, in real world scenarios, replication will be held up for different reasons. Whether it’s table locks, disk I/O, network saturation, or CPU usage, slaves might become several seconds, minutes or even hours behind the master.

In some situations, delays of less than 30 seconds may not be a big issue. Some applications, like social networking applications, would need to have the data match at all times. Lags would not be acceptable.

For example, review this scenario. Let’s say you go to a site and create an account. That would send a write query to the master. Once you’ve finished the account creation, the page will depend on a read query. If the slave is behind the master, it won’t have any data about your new account, and the application will probably tell you that you don’t have an account. That would be pretty annoying for your application’s users.

To check your current lag, simply run SHOW SLAVE STATUS; in MySQL, and review the number following Seconds_Behind_Master. If everything is running well, it should be followed by 0. If NULL is shown, then there is most likely an issue with replication, and you might want to check Last_Error.

So, how can replication lags be corrected? Try these methods:

Review your queries. When queries keep running in MySQL, the slave may be unable to keep up. Make sure that your read queries are as optimized as possible so they complete quickly.

Optimize your MySQL server variables. Be sure to thoroughly review your MySQL configuration for any bottlenecks.

Choose the right storage engines. If you’re making a lot of updates to a table, consider using InnoDB. If your tables are not updated often, consider using MyISAM tables (or even compressed MyISAM tables).

Upgrade your hardware. Find your hardware bottleneck. If it’s the CPU, consider upgrading to a multi-core CPU, or a CPU with a higher clock speed. For I/O bottlenecks, consider a RAID solution with SAS drives. If you’re lucky enough to have a network bottleneck (lucky since it means you’re doing well with CPU and I/O), use a dedicated switch or upgrade to gigabit (or faster) hardware.

MySQL Replication: Horizontal Data Partitioning

major@mhtx.net (Major Hayden) — Mon, 07 Jan 2008 17:57:56 +0000

If you have a master with multiple slaves, you can get some performance and save money on hardware by splitting data horizontally among your servers. For example, if you have one high traffic database and two lower traffic databases, you can selectively split them among the slaves. With five slaves, set up three of the slaves to replicate your high traffic database, and the two other slaves can handle one each out of the two low traffic databases.

This allows you to expand when you’re ready, and you can move your databases around to take advantage of idle servers. MySQL AB already has documentation on how to make this possible.

MySQL Replication: Backups & Data Integrity

major@mhtx.net (Major Hayden) — Fri, 04 Jan 2008 19:39:12 +0000

An often overlooked benefit of MySQL replication is the ability to make reliable backups without affecting the integrity of the MySQL data.

With one MySQL server, backups have a huge impact on the server. If file-based backups are performed, you have to stop MySQL completely while the files are copied (unless you purchase expensive utilities that accomplish this while MySQL is running). If dumps are made with mysqldump, table locking and I/O operations will crush the performance of the server.

You can get around these performance hits by running dumps in single transaction mode, or by restricting mysqldump to locking one table at a time. The performance gain comes at a price, however, as your backups are not a perfect snapshot. After one table is locked for a period of table, previously locked tables are actively changing and some tables might not match up.

By having a slave available, you can perform a snapshot backup and lock all of the tables during the process. This provides an exact point-in-time backup with a very low effect on your MySQL servers’ performance.

MySQL Replication: Redundancy

major@mhtx.net (Major Hayden) — Fri, 04 Jan 2008 02:39:11 +0000

Although performance is a much larger benefit of replication, it provides some redundancy for your application as well. Adding a slave server to a master allows you to perform read operations on either server, but you’re still bound to the master server for writes. In a group of multiple slaves with one master, you have your data available and online in multiple locations, which means that certain servers can fall out of replication without a large disaster.

When disaster does occur, use the following recommendations as a guide.

If the master fails in a two-server replication environment, you will be dead in the water with regards to write queries. You will need to convert the slave into a master. This can be done relatively quickly by following these steps:

Log into MySQL on the slave and run STOP SLAVE; RESET SLAVE;
Add log-bin to the slave’s /etc/my.cnf file and restart MySQL
The slave server will now be running as a master
Adjust your application to send reads and writes to the slave

Once the original master comes back online, set it up just like a new slave. You can skip some steps, such as setting the server-id, since that still should correspond to your overall configuration.

If the master fails in a multiple-server replication environment, you’re still in bad shape for writes. Follow the steps shown above, and then adjust the other slaves (with CHANGE MASTER) so that they pull events from the new master instead.

If a slave fails in any replication environment, adjust your application so that it no longer attempts to send reads to the failed slave. While you work to bring the failed slave back online, your queries will be distributed to the remaining servers.

You can automate many of these operations by using applications like heartbeat, or by using load balancers to automatically route database traffic.

MySQL Replication: Performance

major@mhtx.net (Major Hayden) — Wed, 02 Jan 2008 18:20:05 +0000

MySQL replication can increase performance by allowing developers to spread queries over two servers. Queries that write data must be sent to the master at all times, but queries that read data can be sent to either server. This means that by adding a slave server to a database environment allows you to effectively double your read query performance.

However, there are some large caveats to consider here. The actual web site code itself will need to be written in such a way that read and write queries can be diverted to different destinations. Depending on the size of the application and how it has been developed, the work requires to provide this functionality may be prohibitive for replication.

Some load balancers can balance MySQL query traffic, and this can help if the code cannot balance the load internally. Open source applications like MySQL Proxy and pound can be used as well.

Also, if the queries are not optimized, and the correct storage engines are not used, replication will not work well. If queries take an extended time to execute, the performance gains will be almost non existent. Also, if the wrong storage engines are used, and much of the rows or tables are locked, performance gains will be greatly limited. Some situations may actually cause replication to halt due to locking. When this occurs, the data on the slave becomes stale and SELECTs run against the master and slave will return different results.

In short:

Replication can increase read performance
It cannot fix issues caused by bad queries/storage engines
Write queries can only be sent to the master