Who watches the watcher?

Whilst sharing my Pi over the internet, I discovered a need to hard-reset it from time to time due to the various software which was being tested.

My first thought was some sort of clever microcontroller-based solution for power-cycling the Pi on demand, but there were several technical hurdles and gaps in my knowledge to overcome to implement this solution.

Enter Kripton, who points out that the Broadcom BCM2708 has a hardware watchdog which, with a simple kernel module and watchdog daemon, can automatically reset the Pi when it freezes and even in a variety of other situations.

To get watchdog up and running, you'll first need to:

sudo modprobe bcm2708_wdog
sudo vi /etc/modules
# Add the line "bcm2708_wdog"

Next you need watchdog, a daemon which does little more than send /dev/watchdog a heartbeat every 10 seconds. It can also run arbitrary executables which do things like ping a server, and can trigger reboot when they fail.

sudo apt-get install watchdog chkconfig
chkconfig watchdog on
sudo /etc/init.d/watchdog start

You should also configure watchdog:

sudo vi /etc/watchdog.conf
# Uncomment the line watchdog-device = /dev/watchdog
# You might also want to uncomment max-load-1, or add something like "max-load-1 = 24" to reset your Pi if the load average exceeds 24 in any 1-minute span.

Finally, make sure you've not got any crucial processes running and ensure you deactivate any swap partitions or files, then test your watchdog with a nasty forkbomb:

: (){ :|:& };:

This will ultimately cause a kernel panic and, if everything is set up correctly, your Pi will reboot a few seconds later.

If you're running Arch Linux on your Pi (I've been tinkering under it lately) then these instructions differ, and you'll need to do the following:

Set up the kernel module to start on boot by doing:

echo "bcm2708_wdog" | sudo tee /etc/modules-load.d/bcm2708_wdog.conf

Install the watchdog daemon with:

pacman -S watchdog

And enable/start it with:

sudo systemctl enable watchdog
sudo systemctl start watchdog.service

Arch instructions from: https://blog.ricardoarturocabral.com/2013/01/auto-reboot-hung-raspberry-pi-using-on.html

« Back to index Posted on 2012-08-27 by Philip Howard