What It Actually Takes to Ship Over-the-Air Firmware Updates

Every connected product ships with a quiet promise: we can fix it later. Over-the-air (OTA) updates are how you keep that promise, and that's what makes them different from every other feature. A display bug is a display bug. A bug in the update path takes away your ability to fix any other bug.

Here's what that looks like in practice.

The update that couldn't install its own fix

One release of our Wi-Fi modem firmware had a small parsing bug in the code that asks the cloud whether a new version is available. The reply got cut short, the parser rejected it, and the device concluded there was nothing to download. Every six hours it asked again, and every six hours it decided it was up to date.

We fixed the bug, and the fixed version sat on the server, ready. But units running the broken version couldn't fetch it, because fetching it was exactly the thing that was broken. One field unit kept reporting the old version for three months after the fix shipped.

What got it out was a second path. The host product can tell the modem to update from a specific address over its own command channel, which skips the broken check entirely. Without that, the only option is a technician with a serial cable.

That's the first rule, and the one most teams learn the hard way: an update client can't be trusted to repair itself. Before any release that touches the update code, we test on the bench that the new image can still download and install the next one. And every product keeps a second way in that doesn't depend on the update client being healthy.

The rest of the checklist

Two images, and a way back. A device that loses power halfway through writing new firmware has to wake up on something that runs. That means room for two images, a bootloader that only switches to the new one after checking it's complete and intact, and a rollback if the new image doesn't prove it's healthy after its first boot. It costs flash, so it belongs in the memory budget from the first schematic, not bolted on at the end.

Verify what you wrote, not what the tool reported. During a recent board bring-up, the programming tool reported success while the application area of the chip was still blank. The bootloader found nothing to run, and the only symptom was silence. An exit code of zero means the tool finished. It doesn't mean the chip holds the image. We now erase, write and read back on every flash, and over the air the same job falls to the bootloader: check the image before you run it.

Give every device its own identity. If every unit shares one certificate to talk to the cloud, one extracted key lets someone impersonate or command the whole fleet. Worse, replacing that shared certificate means pushing an update to every device while the old one still has to work, and any unit that misses the window drops off the network. The rule we build to: each device gets its own key at provisioning, and no secret ever ships inside a firmware image. Then one compromised unit is one revoked certificate, not a fleet-wide incident.

Treat message size as part of the protocol. Update commands and the messages around them are often the largest things a device ever receives. We've had a device's network stack quietly drop its cloud connection because one message was larger than its receive buffer, one of the stories in The Seam Problem. Test updates at real sizes, over the real network, on a weak signal.

Know exactly what's in the field. Every image we upload for OTA is built from committed, pushed source, and the upload records the exact source revision. The upload script refuses to run on uncommitted code. When a device reports a version number, we can say precisely what code that is and rebuild it byte for byte.

Cellular makes all of it harder

Cellular devices on LTE-M and NB-IoT add their own constraints. Data is paid for by the byte, coverage comes and goes, and a battery-powered device may sleep most of the day. An update has to survive a dropped connection without starting over, fit inside a data plan, and wait for the device's next wake window. None of that shows up on a bench with strong signal and a USB cable, which is exactly why it has to be tested in the field.

The part nobody budgets for

OTA usually gets scoped as a feature: a download, a flash write, a reboot. In practice it's a system. It spans the bootloader, the memory map, the modem, the cloud service that hosts the images, the production line that gives each unit its identity, and the release process that decides what's safe to ship. In the usual vendor model those belong to several different teams, which makes OTA a textbook seam.

It's also why we build it once, carefully, and reuse it. Our cellular and Wi-Fi modems share one update and cloud platform across products, so a lesson learned on one product protects all of them. The update bug above was fixed once, and the rule it taught us, test that each new image can install the next one, applies to every product we ship.

If OTA is a line item in your product plan, talk to us before the memory map is frozen. It's much cheaper to design in than to retrofit.

OTA is the one feature that has to work when everything else is broken. Build the way back before you need it.