Most linux distributions use some type of mechanism to gracefully stop daemons and unmount storage volumes during a reboot or shutdown. It’s most commonly done via scripts that will wait for each daemon to shut down gracefully before proceeding to the next daemon.
As we know, sometimes servers misbehave due to things put them through, and you can quickly end up in a situation where things are going badly. I’m talking about the type of situation where you’re connected via SSH to a server that controls phone lines for five million people and it sits in a tiny building 400 miles away from the nearest human being. We’re talking bad. If you issue a plain reboot command, it might not even make it that far. Once SSH stops running, you’re going to be out of luck.
If you find yourself in this situation (and I hope you won’t!), you have some options to get your way with a misbehaving server remotely. You can force an immediate reboot with the following:
|
1 2 |
echo 1 > /proc/sys/kernel/sysrq echo b > /proc/sysrq-trigger |
WHOA THERE! This is pretty much the same as pressing the reset button on the server (if equipped). No daemons will be shut down gracefully, no filesystem sync will occur, and you may get the wrath of a fsck (or worse, a non-booting server) upon reboot. To do things a little more carefully, read on.
These are called magic commands, and they’re pretty much synonymous with holding down Alt-SysRq and another key on older keyboards. Dropping 1 into /proc/sys/kernel/sysrq tells the kernel that you want to enable SysRq access (it’s usually disabled). The second command is equivalent to pressing Alt-SysRq-b on a QWERTY keyboard.
There’s a better way of rebooting a misbehaving server that Wikipedia shows with the mnemonic “Reboot Even If System Utterly Broken”:
|
1 2 3 4 5 6 |
unRaw (take control of keyboard back from X), tErminate (send SIGTERM to all processes), kIll (send SIGKILL to all processes), Sync (flush data to disk), Unmount (remount all filesystems read-only), reBoot. |
I can’t vouch for this actually working, but I’m interested to try it. UPDATE: I’ve been told that doing this series of commands with ReiserFS is a very bad idea.
If you want to shut the machine down entirely (please think about it before using this on a remote system):
|
1 2 |
echo 1 > /proc/sys/kernel/sysrq echo o > /proc/sysrq-trigger |
If you want to keep SysRq enabled all the time, you can do that with an entry in your server’s sysctl.conf:
|
1 |
kernel.sysrq = 1 |
Good post!
“I can vouch for this actually working, but I’m interested to try it.” — this sentence doesn’t make much sense.
With modern linuxes, I’ve never had an issue using:
sync
sync
reboot -t 0
Over SSH.
@Kyle: On machines with defunct or dead processes (say a hung NFS mount or disk full problems) “reboot” will not always work. sysrq is a last resort but a welcome one for those rare cases.
One of the useful things to do with this tip, is to use it to test your HA setup (such as with MySQL & DRBD / Heartbeat) worked as expected.
It’s always interesting to know that you got the data back that you were supposed to – and you managed to solve the durability problem that is the ‘D’ in ACID.
Note: you don’t need to turn system request on if you’re echoing into sysrq-trigger, it will always work as long as you are root. You only need to turn it on for key combos.
Thank you! This worked when shutdown -r now, reboot, and init 6 wouldn’t. One of the disks failed and the system was locked up forever trying to unmount it. Fortunately it was just a backup disk and no data was lost on the others.
“With modern linuxes, I’ve never had an issue using:
sync
sync
reboot -t 0
Over SSH.”
I just did! No 5 million user phone switch here, but a remote box at my parents house, I just logged in and it was good and screwed (it’s a bad sign when you see “/bin/ls: I/O error”. Reboot command got me “reboot: command not found”. It seems /dev/sda dropped right off the SATA bus. Of course, I *SERIOUSLY* doubt it’s going to sucessfully reboot (probably, the disk has either crashed, or will need an actual power cycle to show back up) but at least I could attempt a reboot with these instructions, the machine was doing nothing as it stood.
What happens if you cannot access the Website/server remotely, either by SSH or NX….?
Our servers are down in the Core of our TV station, and unless I’m out of town, I COULD drive in and stand in front of the server…So….the server (physical machine) is still running, but I can’t access the Website. Sometimes, can’t log in at all.
Up till now, I’ve just hit the power switch, and gotten yelled at by my Linux mentor….but, if everything seems to be locked up, what do you do? Reset has been unsuccessful….guess there isn’t one on this machine.
If the machine is accessible, at least at the console….standing in front of the machine..and yet my Website is totally inaccessible, non-responsive, what is the proper procedure for restarting the server?
Thanks,
Good input. Thanks!
Man!!! You are my personal Jesus Christ!!!!!
Though my little zombie Linux was only 150 km far.
My zombie was in a different country. It worked like a dream :D
Wow this is really very useful!
I had a linux system in a state – mounted systems hanging. Reboot was doing nothing.
Performed your little trick and voila it was back up and synced in no time. What i see in the logs are
Jul 11 03:28:30 fstab-sync[4289]: removed all generated mount points
Jul 11 03:28:32 fstab-sync[4403]: added mount point /media/cdrom for /dev/hdb
Very tidy!
Hi….im trying to shutdown by just entering ‘shutdown’.But its not working.It says:
shutdown: time expected
Try ‘shutdown –help’ for more information
Can anyone please help?
shutdown requires a time, you want
shutdown nownormally to shutdown immediately, unless you want to give logged in shell users a warning and countdown. If it’s a remote server make sure you have a way to bring the system back up.YOU JUST SAVED MY LIFE! Thank you! In Ohio with a production server in Philadelphia. It came back up!
Thanks for this. I’ll keep this info handy the next time my server says “Shutting down now” after I issue a ‘shutdown -h now’ but nothing happens.
Saved life. Remote server was over loaded, 8 cores, but system load > 20 due to a stupid bug in a program. Could not spawn new process. Executed this magic. Since echo is a shell built-in it did not spawn another process, and saved my life. Thanks.
Thanks,
Been searching for such a command .. It always works.
Thanx Major, this was extremely useful.
Thanks Friend,
one of our production box was not rebooting, from your magic words server rebooted on a fly.
Regards
Basavaraju.M
Brilliant! This little trick just got me out of a pretty bad jam with a remote server that refused to reboot gracefully.
Your a savior!!! The server got in a bad state with the / mount acting up.
Many thanks, command just saved me a trip to the data centre 20 minutes before home time.