Incident response for custom-firmware IoT devices

Connected devices running bespoke firmware are often treated like ordinary endpoints: isolate them, reinstall an image and move on. That approach fails when the firmware itself, its update mechanism or the signing keys have been altered. Custom code can also rely on undocumented services and hardware interfaces, giving attackers persistence that survives a reboot or a factory reset.

For teams in Australia, this matters across very different environments: remote pump stations serving a Sydney water utility, connected medical equipment in a Melbourne hospital and autonomous systems at Pilbara mining sites cannot be restored with the same assumptions or the same urgency. A remote control channel may still be needed for a safety-critical process even after the device is suspected of compromise.

The goal is not simply to bring a device online. It is to restore a known-good state, remove the attacker’s foothold and show that normal operations can resume without carrying the compromise into other systems. That requires coordination between operations, security, firmware engineers and the supplier, particularly when production hardware cannot be physically replaced.

Why custom firmware changes the response

Custom firmware shifts the unit of investigation from an application or operating system to the complete trust chain. A device may combine a vendor build, an integrator’s configuration, locally installed scripts, unsigned update packages and credentials embedded in files or secure storage. If one element is replaced, investigators must determine which parts remain authentic before they trust logs, reset a password or copy an image back.

A stock endpoint can often be compared with a known image. A bespoke device may have no such baseline, especially where each site holds different radio, process or environmental settings. The incident plan should therefore include a trusted reference build, cryptographic hashes, build provenance and a record of every authorised firmware variant. A stolen update-signing key can turn one compromised build server into a fleet-wide supply-chain problem.

The Australian Signals Directorate, based in Canberra, publishes guidance on securing IoT and recommends the Essential Eight as a baseline for organisations. Those controls are a sensible starting point, but a firmware recovery procedure needs a more granular companion: how to validate an image, revoke update keys and operate safely when a device cannot be patched immediately.

Contain the device before chasing the alarm

Containment should reflect what the device can do and what it has learned. Start with an inventory that ties each unit to its owner, location, firmware version, network role, physical capabilities and supplier. Then restrict management access, block risky protocols and remove unnecessary routes to operational networks. A mining gateway with remote-control functions in the Pilbara deserves a firmer boundary than a temperature sensor in the same site, even if both share the same supplier.

Do not power off a device automatically. If it is maintaining a process, shutting it down may create a safety event or destroy volatile evidence. Where there is no immediate physical danger, capture volatile data first, place it on an isolated network or monitoring segment and preserve the ability to observe its communications. Where safety takes priority, document the decision and accept that some evidence may be lost.

Watch for persistence in the update channel. Attackers can install a modified updater, alter a configuration file or introduce a service that survives a normal restart. Blocking inbound connections is insufficient if the device can still receive a malicious package from its update server. In some cases, suspend update credentials or move the device to a quarantined segment while engineers confirm its firmware and boot path.

Preserve evidence and establish the blast radius

Capture what can be trusted, not merely what is convenient. For affected models, record flash contents, bootloader configuration, environment variables, running processes, listening sockets, system logs and local changes. Use write-blocking or a known-good acquisition process where possible, calculate hashes and retain originals offline. Guidance on ransomware and drive recovery explains why protected copies and validated restoration matter when storage is encrypted; the same discipline protects flash images and forensic evidence.

The working hypothesis should cover more than the initially flagged device. Custom firmware may share a certificate, update server, build pipeline or cloud identity across fleets. Review the same devices, credentials, certificates and management systems, and expand the scope when evidence points to a common supplier or distribution point.

Control area Custom-firmware device Conventional managed endpoint
Asset baseline Model, hardware revision, site configuration and firmware hash Standard image, patch level and endpoint agent
Update trust Signing keys, package validation and rollback protection Managed update service and signed policy
Identity Unique device and service credentials Central user, device and directory identities
Evidence Flash image, boot chain and local logs Disk image, event logs and cloud telemetry
Recovery Supplier-supported restore or hardware replacement Rebuild from a known-good image
Ongoing control Firmware monitoring and physical access controls EDR, vulnerability management and endpoint isolation

In an Australian operations room, “all sorted” after a visual check is not evidence of recovery. Identity deserves particular attention because one reused service account may operate many devices. A recovery guide for domain controller credential recovery offers a useful parallel with golden ticket attacks: once privileged identity material is exposed, rebuilding individual endpoints is not enough. Rotate dependent secrets, invalidate tickets where applicable and verify every system that reused the compromised identity.

Recover systems without restoring the compromise

Recovery begins with an immutable reference: a factory image or reviewed firmware build tied to a specific hardware revision. Validate it on a representative device, confirm the expected boot chain and compare security settings with the production baseline. Rolling the old build back onto every unit can reintroduce the weakness if the vulnerable component or stolen signing key remains.

Privileged identity recovery must run alongside device restoration. The practical steps in domain controller credential recovery are relevant even where no domain controller exists, because an attacker may have harvested reusable keys or credentials from the device. Rotate those secrets in dependency order, revoke trust paths and watch for re-entry after the first fleet reconnects.

A staged rollout beats a mass deployment. Restore one device, exercise its functions, observe its traffic and then release a small cohort. Keep quarantined units available for comparison and maintain a rollback point that has been tested rather than assumed. If safe operation requires the original device, replace the hardware, revoke its certificates and treat any configuration exported from it as potentially hostile.

Harden updates, identities and monitoring

The lessons from an incident should change the build and operating model. Require signed firmware, anti-rollback protection, secure boot and authorised over-the-air update channels. Use a hardware root of trust where the design permits it, give every device a unique identity, avoid embedded shared passwords and keep signing keys in a controlled environment with rotation and usage records. A software bill of materials helps identify affected components, but it must be tied to each firmware variant rather than a generic file.

Detection also needs to understand device behaviour. Baseline expected DNS, process activity, outbound connections and update timing, then alert on changes such as a new certificate, an altered partition hash or a device contacting a previously unseen host. Store logs centrally where practical, with time synchronisation and tamper resistance. Local logs alone may disappear with the compromised unit, while retained logs can show how far the intrusion spread.

Australian reporting and risk obligations should be included in the plan. The Australian Signals Directorate’s Australian Cyber Security Centre provides reporting guidance, while APRA’s CPS 234 requires relevant Australian financial entities to maintain and test information-security controls. Connected payment systems or operational technology supplied to a covered entity can bring supplier responsibilities into that framework, so contracts should specify patching, notification and forensic support.

Build a repeatable response for connected operations

Turn the procedure into rehearsed playbooks for distinct device classes. A water utility in Sydney may need to isolate a remote pump controller without interrupting pressure management, while a Melbourne hospital may have to maintain strict availability for connected medical devices. Each playbook should nominate decision owners, engineering contacts, safe shutdown criteria, evidence locations and criteria for declaring a device trustworthy.

Vendor access is a recurring weak point. Custom firmware often comes with remote engineering tools and shared support accounts. Use individually attributable accounts, time-limited privileged access, session recording and a change record that links every authorised firmware build to its approver. A supplier should be able to provide the signing-key inventory, affected-version list and chain of custody for updates, even when an incident spans several product lines.

A coordinated platform can help organisations bring vendor findings, threat intelligence and remediation work into one operating process. CARM Security’s ecosystem is relevant to this need because it supports post-breach containment and mitigation across multiple security technologies. The platform does not replace a firmware inventory or an engineering runbook; it can make the wider response visible while the internal team owns technical recovery.

The practical takeaway is to maintain a verified recovery image, unique device identity and tested isolation path for every custom-firmware class, then rehearse restoration under realistic operating constraints. Treat those artefacts as operational assets that must be available during an incident, rather than documentation that can wait for a quieter week.