Most linux distributions use some type of mechanism to gracefully stop daemons and unmount storage volumes during a reboot or shutdown. It’s most commonly done via scripts that will wait for each daemon to shut down gracefully before proceeding to the next daemon.
As we know, sometimes servers misbehave due to things put them through, and you can quickly end up in a situation where things are going badly. I’m talking about the type of situation where you’re connected via SSH to a server that controls phone lines for five million people and it sits in a tiny building 400 miles away from the nearest human being. We’re talking bad. If you issue a plain reboot command, it might not even make it that far. Once SSH stops running, you’re going to be out of luck.
If you find yourself in this situation (and I hope you won’t!), you have some options to get your way with a misbehaving server remotely. You can force an immediate reboot with the following:
|
1 2 |
echo 1 > /proc/sys/kernel/sysrq echo b > /proc/sysrq-trigger |
WHOA THERE! This is pretty much the same as pressing the reset button on the server (if equipped). No daemons will be shut down gracefully, no filesystem sync will occur, and you may get the wrath of a fsck (or worse, a non-booting server) upon reboot. To do things a little more carefully, read on.
These are called magic commands, and they’re pretty much synonymous with holding down Alt-SysRq and another key on older keyboards. Dropping 1 into /proc/sys/kernel/sysrq tells the kernel that you want to enable SysRq access (it’s usually disabled). The second command is equivalent to pressing Alt-SysRq-b on a QWERTY keyboard.
There’s a better way of rebooting a misbehaving server that Wikipedia shows with the mnemonic “Reboot Even If System Utterly Broken”:
|
1 2 3 4 5 6 |
unRaw (take control of keyboard back from X), tErminate (send SIGTERM to all processes), kIll (send SIGKILL to all processes), Sync (flush data to disk), Unmount (remount all filesystems read-only), reBoot. |
I can’t vouch for this actually working, but I’m interested to try it. UPDATE: I’ve been told that doing this series of commands with ReiserFS is a very bad idea.
If you want to shut the machine down entirely (please think about it before using this on a remote system):
|
1 2 |
echo 1 > /proc/sys/kernel/sysrq echo o > /proc/sysrq-trigger |
If you want to keep SysRq enabled all the time, you can do that with an entry in your server’s sysctl.conf:
|
1 |
kernel.sysrq = 1 |
Thank you for this page, I managed to reboot my VPS thanks to hints within, even though I did not did not need to resort to the sysrq method.
it says here that we can do this by entering the command when the ssh stops running.So does it mean that without using putty we can enter to the system and reboot it remotely ??
Can you tell me where to enter the command?
I tried this but I couldn’t get my FS or data to corrupt. I think it will still let the HDD or SSD flush what it has. Perhaps power saving options can force a hard power cycle of storage media.
Thanks a lot. I was able to login remotely but had lots of i/o errors and a normal reboot command didn’t work. That worked like a charm and saved me from trouble with having to ask for a manual reset from the provider.
U’re a life saver !
Thanks a lot ;)
Thanks a lot :) it help
Nice Article. How it help to developer in terms of balance the day to day life.
Thanks much, restarted a i/o mess of remote debian system. Great stuff!
i just restart my linux server and it don’t respond, maybe it hang somewhere in boot menu, can someone help me how remotely reboot my linux. Thanks i will wait for fast reply.
those are things that help if in a hurry
Thanks Major. Great write up.
You’re welcome, Jesse! :)
Part of a global team at work – where the devs are currently asleep on the other side of the world. Move to a new version of linux, with a new toolset has seen dramas where the highly customised and optimised emulation software in use sh**s the bed on the new hardware and operating system.
With limited knowledge of the iLO tech in use by foreign team, and seeing things like ps ax hang, I knew a graceful shutdown just wasn’t going to work, and then I’d lose access to the console – using this sysrq functionality helped me reboot the physical system underlying the emulator without needing to rely on iLO and meant that I had the system back up in minutes rather than having to go hunting for a manual on java whispering in modern OSs (iLO folks need to update!)
I’d heard of, and used the physical keys for sysrq before, but never done it through the command line.
Thanks for your useful and informative article.
Awesome! I’m glad I could help. ;)
Hi Mr. Major, sometime the centos 7 server has been block and I couldn’t do nothing, only press the reset bottom. So I searched some reasons in internet and I found that I will execute the next command:
Know the sysrq estate:
# grep SYSRQ /boot/config-$(uname -r)
CONFIG_MAGIC_SYSRQ=y ——-> Is enable
# cat /proc/sys/kernel/sysrq
Result:
16 – enable sync command
Will you explain me what those meaning and how can I do?
Thaks very much.
Reagards
Works great! Thank you very much for sharing.