Detect Lockups | Articles

If an Arduino program crashes in the forest, will it make a sound? Unfortunately not. No blue screen of death, nor jarring squawk will alert you to the program’s doom. It will typically just quietly and unobtrusively stop running whatever robot, keypad or front door it is supposed to be looking after. In this article, I’ll show you how to use the watchdog timer to help track down the bug causing the lockup. But first, there’s a quick review of the watchdog timer itself for those new to the game.

The Arduino Watchdog Timer

Most microcontrollers, including the Atmel chips using on the Arduino, have a watchdog timer. Its a sort of policeman that resets the micro if it thinks the program has stopped working properly. You signal the program is working correctly simply by calling wdt_reset() each time through the loop, after enabling the watchdog in the setup routine. So although the following program will get stuck in the while loop, the watchdog timer will reset the micro every second and the program will do at least part of its job. Of course it is better to fix the bug causing the lockup. And for that we can use the two stage watchdog timer: interrupt and reset.

#include <avr/wdt.h>

void setup()

{

Serial.begin(9600);

wdt_enable(WDTO_1S);

}

void loop()

{

// the program is alive...for now.

wdt_reset();

Serial.println("Hello");

while (1)

; // do nothing. the program will lockup here.

Serial.println("Can't get here");

}

A word of caution here: there is a bug in the early Mega2560 bootloader. If you turn on the watchdog timer on this board, it can get stuck in the bootloader. Update the bootloader using Arduino IDE version 1.0.4 (or later) and all should be well.

Watchdog Interrupt and Reset

The watchdog timer on the Arduino microcontrollers can operate in four different modes:

Disabled: the program simply gets stuck if it runs into an infinite loop
Reset: if wdt_reset() is not called within every timeout period, the watchdog timer will reset the program and it will start again from the beginning
Interrupt: an interrupt will be generated if wdt_reset() isn’t called within every timeout period
Interrupt + reset: a combination of the last two. If wdt_reset() isn’t called within the timeout period, the watchdog timer will first raise an interrupt. Then, if wdt_reset() isn’t called within another timeout period, the program will reset and start again

It is the two interrupt modes that are useful for detecting lockups. Why? When an interrupt is fired the micro pushes the current program counter onto the stack. That’s the address where the Arduino was executing code when the watchdog fired…and a good place to look for the bug. Of course, if the program has just crashed it is going to be a bit risky trying to send the address out the serial port. Instead, the address can be saved in the eeprom and printed next time the program starts.

An Arduino Crash Location Detection Example

We packaged all this up into a simple example, which is available on github. The program will work with the Arduino IDE, or our Visual Studio Arduino build tool and it is made up of three key files:

ApplicationMonitor.h/ ApplicationMonitor.cpp: this is the core part of the implementation. It handles the watchdog interrupt, storing the information to the EEPROM and printing diagnostic information at startup.
Program.cpp: an example program which locks up after a few iterations to illustrate using the library.

The example program shows how the crash handler could be used in a typical program. The ApplicationMonitor.h and ApplicationMonitor.cpp files could be included in your program folder, as here, or moved into a library folder to be used across several programs. The whole test program is reproduced below, but the key lines are:

Include the crash monitor header file.

#include "ApplicationMonitor.h"

1

#include "ApplicationMonitor.h"
Create a global monitor object. This must be called ApplicationMonitor otherwise you will get errors when you build the program. You can optionally set the eeprom address where data is saved and the number of crash reports that will be saved here. By default, 10 reports are saved, starting from address 500.

Watchdog::CApplicationMonitor ApplicationMonitor;

1

Watchdog::CApplicationMonitor ApplicationMonitor;
Write the crash data out the serial port, at a convenient location:

ApplicationMonitor.Dump(Serial);

1

ApplicationMonitor.Dump(Serial);
Initialize the application monitor and set the timeout. This would normally live in the setup function, as here:

ApplicationMonitor.EnableWatchdog(Watchdog::CApplicationMonitor::Timeout_4s);

1

ApplicationMonitor.EnableWatchdog(Watchdog::CApplicationMonitor::Timeout_4s);
Keep calling the IAmAlive function within the timeout period to prevent automatic reset by the watchdog timer. This would normally happen at the top of the loop function.

ApplicationMonitor.IAmAlive();

1

ApplicationMonitor.IAmAlive();
Finally, you can save a 32-bit value with the next crash report. This could record any information that might be helpful to tracking down the bug such as the current program mode.

ApplicationMonitor.SetData(g_nIterations++);

1

ApplicationMonitor.SetData(g_nIterations++);

The complete example program:

#include "ApplicationMonitor.h"

Watchdog::CApplicationMonitor ApplicationMonitor;

// An LED to flash while we can. 
const int gc_nLEDPin = 13;

// countdown until the program locks up. 
int g_nEndOfTheWorld = 15;

// number of iterations completed. 
int g_nIterations = 0;

void setup()
{
  Serial.begin(9600);
  pinMode(gc_nLEDPin, OUTPUT);
  Serial.println("Ready");

ApplicationMonitor.Dump(Serial);
  ApplicationMonitor.EnableWatchdog(Watchdog::CApplicationMonitor::Timeout_4s);
  //ApplicationMonitor.DisableWatchdog();

Serial.println("Hello World!");
}

void loop()
{
  ApplicationMonitor.IAmAlive();
  ApplicationMonitor.SetData(g_nIterations++);

Serial.println("The end is nigh!!!");

digitalWrite(gc_nLEDPin, HIGH);   // turn the LED on (HIGH is the voltage level)
  delay(200);               // wait for a second
  digitalWrite(gc_nLEDPin, LOW);    // turn the LED off by making the voltage LOW
  delay(200);               // wait for a second

if (g_nEndOfTheWorld == 0)
  {
    Serial.println("The end is here. Goodbye cruel world.");
    while(1)
      ; // do nothing until the watchdog timer kicks in and resets the program. 
  }

--g_nEndOfTheWorld;
}

#include "ApplicationMonitor.h"

Watchdog::CApplicationMonitor ApplicationMonitor;

// An LED to flash while we can.

const int gc_nLEDPin = 13;

// countdown until the program locks up.

int g_nEndOfTheWorld = 15;

// number of iterations completed.

int g_nIterations = 0;

void setup()

{

Serial.begin(9600);

pinMode(gc_nLEDPin, OUTPUT);

Serial.println("Ready");

ApplicationMonitor.Dump(Serial);

ApplicationMonitor.EnableWatchdog(Watchdog::CApplicationMonitor::Timeout_4s);

//ApplicationMonitor.DisableWatchdog();

Serial.println("Hello World!");

}

void loop()

{

ApplicationMonitor.IAmAlive();

ApplicationMonitor.SetData(g_nIterations++);

Serial.println("The end is nigh!!!");

digitalWrite(gc_nLEDPin, HIGH); // turn the LED on (HIGH is the voltage level)

delay(200); // wait for a second

digitalWrite(gc_nLEDPin, LOW); // turn the LED off by making the voltage LOW

delay(200); // wait for a second

if (g_nEndOfTheWorld == 0)

{

Serial.println("The end is here. Goodbye cruel world.");

while(1)

; // do nothing until the watchdog timer kicks in and resets the program.

}

--g_nEndOfTheWorld;

}

Here’s a screenshot of the MegunoLink Pro monitor window after the program has been running for a minute or two showing the program output:

You can see it has saved 9 crash reports so far, and the next one will be saved at location 9 (of 10). The interesting part, the location in our program where the lockup occurred, is included too. For crash 8, the program was executing instructions at program memory address 0x398. For tracking down the offending line of code in the program, we need the byte address though. In this case: 0x730 (the byte address is simply twice the program memory address).

The hunting of the source code

By itself, the program memory address is not very useful. It isn’t a line number of file name; it is simply the location in program memory of the instruction that the Arduino was executing when the lock-up occurred. To get from the program memory address, to the offending line of source code we need to use the disassembler. To upload your program to an Arduino, the IDE first compiles the code in each file into instructions the microcontroller can understand—assembly code—creating a separate object file for each source file. A linker joins these together into an executable and linking format, or ELF, file. The ELF file contains the instructions at each program address and the disassembler lets us connect those addresses back to the original program code using this magical incantation:

1	avr-objdump -d -S -j .text CrashTracking.elf > Disassembly.txt

Here, CrashTracking.elf is the linker output for the crashing program and avr-objdump is part of the Arduino installation that you’ll find in “arduino\hardware\tools\avr\bin”. You’ll need to add this folder to your system path to run the object dump program. Running this command will create a text file named “Disassembly.txt”. The disassembly file contains the original source code mixed in with the assembly instructions, as in the excerpt below. Lines with assembly instructions start with the program counter address. Typically, you’ll see a line of source code followed by the assembly instructions that the microcontroller executes to implement the source line.

if (g_nEndOfTheWorld == 0)

718: 80 91 4c 02 lds r24, 0x024C

71c: 90 91 4d 02 lds r25, 0x024D

720: 00 97 sbiw r24, 0x00 ; 0

722: 39 f4 brne .+14 ; 0x732 <loop+0x82>

{

Serial.println("The end is here. Goodbye cruel world.");

724: 8b e8 ldi r24, 0x8B ; 139

726: 94 e0 ldi r25, 0x04 ; 4

728: 63 e1 ldi r22, 0x13 ; 19

72a: 72 e0 ldi r23, 0x02 ; 2

72c: 0e 94 aa 09 call 0x1354 ; 0x1354 <_ZN5Print7printlnEPKc>

730: ff cf rjmp .-2 ; 0x730 <loop+0x80>

while(1)

; // do nothing until the watchdog timer kicks in and resets the program.

}

--g_nEndOfTheWorld;

732: 01 97 sbiw r24, 0x01 ; 1

734: 90 93 4d 02 sts 0x024D, r25

738: 80 93 4c 02 sts 0x024C, r24

73c: 08 95 ret

So to find the offending lock-up, search through the disassembly until you find an assembly line with the same address as the byte address in the application monitor output. In this case, the lock-up occurs at address 0x730, or line 13 in the listing above. Ah-ha! The problem is obvious: address 0x730 simply jumps back to itself in an endless loop that will end only with the collapse of the Universe (or the power plug is pulled, whichever comes first). But you don’t need to read the assembly code to make use of the application monitor. For line 14 and 15 have the corresponding program code (sometimes the program code comes after the program address): an endless while loop keeping the Arduino busy doing absolutely nothing. Remove this line and the program will cheerfully, like all soothsayers, predict a doom that never arises.

The end (of all bugs)

Okay, perhaps this won’t track down every bug in your program, but if you think this will be a useful tool in your quest for the perfect program, post a comment below. You can download the Arduino lockup monitor from github. And download MegunoLink Pro too. Once you’ve fixed the bugs in your program, you can use MegunoLink Pro to monitor, plot or log your data in real-time or build a simple user interface to control your Arduino program from your PC.

Arduino watchdog, lockups

The Arduino Watchdog Timer

Watchdog Interrupt and Reset

An Arduino Crash Location Detection Example

The hunting of the source code

The end (of all bugs)

Leave a Comment Cancel reply

Leave a Comment
Cancel reply