Testing Firmware and Handling Failures

Firmware mixes logic that could run anywhere with code that only makes sense on the hardware. Testing works best when each kind is tested where it is cheapest:

Unity is a unit-test framework for C built with embedded toolchains in mind: a single C file and two headers that compile with any compiler, so the same tests can run on a PC and, printing over a UART, on the target. A test file supplies setUp() and tearDown(), and main() runs each test with RUN_TEST between UNITY_BEGIN() and UNITY_END().

Code that reads flash addresses and registers directly can only be tested on the target. Code that takes data and returns a decision can be tested anywhere. Split the settings loader so that the decision is a pure function:

#include <string.h>
#include "unity.h"
#include "settings.h"   /* const record_t *settings_pick(const record_t *a, const record_t *b); */
#include "test_helpers.h" /* record_t make_record(uint32_t seq); */

void setUp(void) {}
void tearDown(void) {}

static void test_newer_valid_slot_wins(void) {
    record_t a = make_record(5), b = make_record(6);      /* test helper: fills in a valid CRC */
    TEST_ASSERT_TRUE(settings_pick(&a, &b) == &b);
}
static void test_corrupt_newer_slot_is_ignored(void) {
    record_t a = make_record(5), b = make_record(6);
    b.payload[0] ^= 0x01;                                  /* the CRC no longer matches */
    TEST_ASSERT_TRUE(settings_pick(&a, &b) == &a);
}
static void test_both_erased_gives_defaults(void) {
    record_t a, b;
    memset(&a, 0xFF, sizeof a); memset(&b, 0xFF, sizeof b);
    TEST_ASSERT_NULL(settings_pick(&a, &b));               /* the caller loads defaults */
}
static void test_sequence_wraps(void) {
    record_t a = make_record(0xFFFFFFFFu), b = make_record(0);
    TEST_ASSERT_TRUE(settings_pick(&a, &b) == &b);
}

int main(void) {
    UNITY_BEGIN();
    RUN_TEST(test_newer_valid_slot_wins);
    RUN_TEST(test_corrupt_newer_slot_is_ignored);
    RUN_TEST(test_both_erased_gives_defaults);
    RUN_TEST(test_sequence_wraps);
    return UNITY_END();
}

The flash access itself goes behind a small interface (read, erase, program) with two implementations: the real driver on the target and a fake in a RAM array on the host.

A passing test suite proves only that the code does what the tests check. One way to measure a suite is to plant plausible bugs in copies of the code and see whether any test fails; a bug that survives every test marks a gap. This is mutation testing, and it can be automated.

↑ This step uses the figure at the top of the page.

The figure makes the point for settings_pick(). Tests of the normal case, where both slots are valid, pass on almost every buggy copy. The bugs that matter live at the boundaries: a corrupt slot, both slots erased, a sequence number that wraps from 0xFFFFFFFF to 0. The test cases worth writing are the ones power loss, noise and time actually produce.

The fake flash turns power loss into a test input. Make it count program and erase operations and “lose power” during operation $n$, leaving that operation half done (ideally with random partial bit states). Then, for every $n$ from 0 to the total number of operations in a save:

1. start from a known good state,
2. run the save with power failing after operation $n$,
3. run the start-up code on what remains, and
4. assert that it recovers either the old settings or the new ones, never defaults and never garbage.

That checks every possible interruption point, which no amount of plugging and unplugging on the bench can do. The same approach injects other faults: a sensor that returns an error, a bus that times out, a queue that is full.

A sensor read over I²C times out after 10 ms when it fails, and fails transiently about 1 % of the time. The control loop can afford 100 ms per read.

With 2 retries and a 1 ms backoff that doubles, the probability that all three attempts fail is $0.01^{3} = 10^{-6}$, if failures are independent. The worst case is $3 \times 10 + 1 \times (2^{2} - 1) = 33$ ms, inside the budget.

With a 25 ms timeout, 3 retries and a 5 ms first backoff, $P_{\text{fail}} = 0.01^{4} = 10^{-8}$, but the worst case is $4 \times 25 + 5 \times (2^{3} - 1) = 135$ ms, over the budget. And if the sensor is unplugged, every read costs that full worst case, so the loop misses its deadline on every pass until the firmware stops retrying.

The independence assumption matters: a bus held low by a crashed device fails every attempt, and $p^{n+1}$ says nothing about it.