Linux: emergency reboot or shutdown with magic commands

Most linux distributions use some type of mechanism to gracefully stop daemons and unmount storage volumes during a reboot or shutdown. It’s most commonly done via scripts that will wait for each daemon to shut down gracefully before proceeding to the next daemon.

As we know, sometimes servers misbehave due to things put them through, and you can quickly end up in a situation where things are going badly. I’m talking about the type of situation where you’re connected via SSH to a server that controls phone lines for five million people and it sits in a tiny building 400 miles away from the nearest human being. We’re talking bad. If you issue a plain reboot command, it might not even make it that far. Once SSH stops running, you’re going to be out of luck.

If you find yourself in this situation (and I hope you won’t!), you have some options to get your way with a misbehaving server remotely. You can force an immediate reboot with the following:

echo 1 > /proc/sys/kernel/sysrq 
echo b > /proc/sysrq-trigger

WHOA THERE! This is pretty much the same as pressing the reset button on the server (if equipped). No daemons will be shut down gracefully, no filesystem sync will occur, and you may get the wrath of a fsck (or worse, a non-booting server) upon reboot. To do things a little more carefully, read on.

These are called magic commands, and they’re pretty much synonymous with holding down Alt-SysRq and another key on older keyboards. Dropping 1 into /proc/sys/kernel/sysrq tells the kernel that you want to enable SysRq access (it’s usually disabled). The second command is equivalent to pressing Alt-SysRq-b on a QWERTY keyboard.

There’s a better way of rebooting a misbehaving server that Wikipedia shows with the mnemonic “Reboot Even If System Utterly Broken”:

unRaw      (take control of keyboard back from X),  
 tErminate (send SIGTERM to all processes),
 kIll      (send SIGKILL to all processes), 
  Sync     (flush data to disk),
  Unmount  (remount all filesystems read-only),
reBoot.

I can’t vouch for this actually working, but I’m interested to try it. UPDATE: I’ve been told that doing this series of commands with ReiserFS is a very bad idea.

If you want to shut the machine down entirely (please think about it before using this on a remote system):

echo 1 > /proc/sys/kernel/sysrq 
echo o > /proc/sysrq-trigger

If you want to keep SysRq enabled all the time, you can do that with an entry in your server’s sysctl.conf:

kernel.sysrq = 1

Comments

  1. W says

    “I can vouch for this actually working, but I’m interested to try it.” — this sentence doesn’t make much sense.

  2. chrisd says

    @Kyle: On machines with defunct or dead processes (say a hung NFS mount or disk full problems) “reboot” will not always work. sysrq is a last resort but a welcome one for those rare cases.

  3. says

    One of the useful things to do with this tip, is to use it to test your HA setup (such as with MySQL & DRBD / Heartbeat) worked as expected.

    It’s always interesting to know that you got the data back that you were supposed to – and you managed to solve the durability problem that is the ‘D’ in ACID.

  4. vinsane says

    Note: you don’t need to turn system request on if you’re echoing into sysrq-trigger, it will always work as long as you are root. You only need to turn it on for key combos.

  5. Ac says

    Thank you! This worked when shutdown -r now, reboot, and init 6 wouldn’t. One of the disks failed and the system was locked up forever trying to unmount it. Fortunately it was just a backup disk and no data was lost on the others.

  6. hwertz says

    “With modern linuxes, I’ve never had an issue using:

    sync
    sync
    reboot -t 0

    Over SSH.”
    I just did! No 5 million user phone switch here, but a remote box at my parents house, I just logged in and it was good and screwed (it’s a bad sign when you see “/bin/ls: I/O error”. Reboot command got me “reboot: command not found”. It seems /dev/sda dropped right off the SATA bus. Of course, I *SERIOUSLY* doubt it’s going to sucessfully reboot (probably, the disk has either crashed, or will need an actual power cycle to show back up) but at least I could attempt a reboot with these instructions, the machine was doing nothing as it stood.

  7. says

    What happens if you cannot access the Website/server remotely, either by SSH or NX….?

    Our servers are down in the Core of our TV station, and unless I’m out of town, I COULD drive in and stand in front of the server…So….the server (physical machine) is still running, but I can’t access the Website. Sometimes, can’t log in at all.

    Up till now, I’ve just hit the power switch, and gotten yelled at by my Linux mentor….but, if everything seems to be locked up, what do you do? Reset has been unsuccessful….guess there isn’t one on this machine.

    If the machine is accessible, at least at the console….standing in front of the machine..and yet my Website is totally inaccessible, non-responsive, what is the proper procedure for restarting the server?

    Thanks,

  8. tsoft says

    Man!!! You are my personal Jesus Christ!!!!!
    Though my little zombie Linux was only 150 km far.

  9. says

    Wow this is really very useful!
    I had a linux system in a state – mounted systems hanging. Reboot was doing nothing.
    Performed your little trick and voila it was back up and synced in no time. What i see in the logs are

    Jul 11 03:28:30 fstab-sync[4289]: removed all generated mount points
    Jul 11 03:28:32 fstab-sync[4403]: added mount point /media/cdrom for /dev/hdb

    Very tidy!

  10. Su says

    Hi….im trying to shutdown by just entering ‘shutdown’.But its not working.It says:
    shutdown: time expected
    Try ‘shutdown –help’ for more information
    Can anyone please help?

  11. E71 says

    Thanks for this. I’ll keep this info handy the next time my server says “Shutting down now” after I issue a ‘shutdown -h now’ but nothing happens.

  12. says

    Saved life. Remote server was over loaded, 8 cores, but system load > 20 due to a stupid bug in a program. Could not spawn new process. Executed this magic. Since echo is a shell built-in it did not spawn another process, and saved my life. Thanks.

Trackbacks

  1. [...] I was doing some troubleshooting on a server and, as a joke, a friend sends me an article to the magic shutdown commands for a solution. I found the article interesting and, once I got home, proceeded to blow up my [...]

Leave a Reply

Your email address will not be published. Required fields are marked *