Managing incident response when primary systems go dark
Picture the scene: a Tuesday afternoon in Sydney, your security operations centre is humming, tickets are flowing in, and then the ticketing platform disappears. Email follows, then single sign-on, then your SIEM. The very systems built to coordinate the response become unavailable, and the team is suddenly making decisions over mobile phones and shared documents. This is not a hypothetical — it is the shape of a successful ransomware or wiper event, where adversaries routinely target the responder's own infrastructure alongside the victim environment. When primary systems are down, the discipline of incident response becomes less about dashboards and more about rehearsed muscle memory.
Across Australia, recent years have shown just how often those primary systems are weaponised against the defenders themselves. The DP World Australia incident in late 2023 stalled container movement across Melbourne, Sydney, Brisbane, and Fremantle and offered a sharp reminder that operational technology and IT are tangled together. When corporate identity, email, and SOC tooling all sit on the same estate that is being encrypted, an enterprise needs a different playbook. CARM supports organisations through exactly that window — post-breach attack remediation and mitigation that continues regardless of whether the usual dashboards are still online.
Building a sovereign response capability before the lights go out
Pre-incident readiness is the difference between confusion and choreography. Australian security leaders are increasingly being asked to evidence this against the Security of Critical Infrastructure (SOCI) Act's obligations for critical infrastructure entities, and against APRA's CPS 234 for financial services. Both regimes expect documented, exercised incident response plans. That documentation needs to answer a question often skipped: what happens when the plan itself depends on technology that is no longer available?
A practical answer starts with naming your out-of-band channels in advance. SOC leads across the country will routinely tell you the most useful investment they made in the last year was a printed contact list and a cloud document store that cannot be locked out by the same compromise. The paragraphs below name the assets that need to be staged.
Out-of-band assets to pre-stage
- A conferencing bridge hosted by a provider different from your primary unified communications stack, with bridge numbers printed and physically distributed.
- A cloud document store in a separate identity and network boundary, holding runbooks, contact trees, and credential recovery procedures.
- A crisis contact list maintained as a printed copy held at operations and security leadership homes, plus a read-only PDF in the secondary store.
- A pre-arranged escalation memo that already lists the ACSC Cyber Security Hotline on 1300 CYBER1, your cyber insurer's loss-adjustment line, and your forensic retainer.
For Australian SOCs, sovereign redundancy matters more than the headlines suggest. Hosting a backup collaboration environment in AWS Sydney, or in Microsoft's Azure Australia Central for security-cleared workloads, gives you a geographically distant channel that is unlikely to be taken out by the same incident. Pair this with regular exercises — at least twice a year — where you simulate the loss of identity, email, and ticketing simultaneously. Teams that have done this find the actual incident clinical; teams that have not find it chaotic.
Operating without identity, email, and ticketing
When Active Directory, Entra ID, Okta, or any central identity provider is offline, every workflow that requires a login is broken. Containment actions frequently need to be executed on systems you can no longer authenticate to in the normal way. The Australian market has matured on this point: the major banks — CBA, NAB, ANZ, and Westpac — alongside the telcos and the ASX itself, all maintain documented break-glass procedures for precisely this reason.
Pre-staged credentials are the practical answer.
Hardware and credentials to stage in advance
- Two FIDO2 hardware security keys per privileged administrator, stored at separate physical sites — one at home, one sealed at a branch office.
- Printed MFA recovery codes held in fire-resistant envelopes at multiple locations, with two-person access controls.
- A break-glass local account on every privileged system, with documented ownership and a password kept only in the secondary document store.
- A separate collaboration tenant on a different provider, used only during declared incidents and governed by out-of-band moderation.
Privileged access management platforms such as CyberArk, BeyondTrust, and Delinea all offer emergency local account access that does not depend on central directory reachability, but only if the underlying vault is hosted on infrastructure that is not part of the in-scope estate. Email is the second big disappearance: a moderated distribution list on a separate provider, kept lean and rarely used, becomes the channel that executives learn to monitor alongside the corporate tenant. Practise it — people forget channels they do not touch.
Triage without telemetry: the fallback data sources
When the SIEM is offline, triage shifts to whatever is still reaching out: EDR agents on endpoints, network taps on switches, cloud-native audit logs, and DNS resolvers that pass traffic through external services. The goal at this stage is to bound the blast radius, not to rebuild a full picture. A simple rule helps: if the system still phones home to a vendor cloud, it is a usable signal.
| Data source | Survives identity outage | Survives on-prem encryption | Survives cloud control plane loss | Best use during degraded response |
|---|---|---|---|---|
| EDR cloud console (CrowdStrike, SentinelOne, Defender for Endpoint, Sophos, Trellix) | Yes | Yes | Mostly | Endpoint isolation, kill process, retrieve artefacts |
| Network NetFlow or sFlow on switches and routers | Yes | Yes | Yes | Lateral movement detection, beaconing visualisation |
| DNS resolver logs (forwarders, Umbrella, Quad9, 1.1.1.1) | Yes | Yes | Yes | C2 beaconing, DNS tunnelling identification |
| Email journaling to a third-party archive | Yes | Mostly | Yes | Phishing source identification, recipient mapping |
| Cloud provider audit logs (AWS CloudTrail, Azure Activity Log) | Yes | Yes | No | Identity administrative actions, resource changes |
The table is worth reading twice. Notice the final row is the only one that fails when the cloud control plane itself is affected. A 2024 SaaS outage affecting several Australian customers demonstrated exactly this trade-off: when the productivity platform went offline, customers lost both the application and the audit trail of who had been using it.
EDR consoles keep the clearest picture because their agents continue to phone home to vendor data centres during a directory outage. From those consoles an IR team can isolate hosts, retrieve memory or disk artefacts for digital forensics, and trigger scripted containment actions. Network telemetry provides east-west visibility that does not require identity to interpret, and it survives the loss of every application server in the datacentre. DNS resolver logs expose command-and-control beaconing even when authentication is broken — one of the few signal sources that an adversary cannot easily silence without taking the host off the internet entirely.
Communications when your own channels are compromised
The Australian regulatory environment around incident communications is unusually dense. A notifiable data breach under the Privacy Act 1988 requires assessment within 30 days and notification to the OAIC and to affected individuals "as soon as practicable" once serious harm is likely. For financial services, APRA's CPS 234 requires notification of material incidents to APRA "as soon as possible" and within 72 hours of determination. Critical infrastructure entities under the SOCI Act must report cyber incidents to the ASD via cyber.gov.au, generally within 30 days but as early as 72 hours for incidents of national significance.
When corporate email, Microsoft Teams, and Slack are down, the people you need to reach are regulators, executives, customers, the media, and the broader staff body. Each of these groups needs its own channel and cadence. A pre-approved relationship with a communications firm — common among ASX-listed companies and the larger customer-owned banking institutions — shortens the time to a published holding statement from hours to minutes. A standing retainer with a forensic and IR provider such as Mandiant, CrowdStrike Services, NCC Group, or local firms such as CyberCX or Sentaris gives you a partner whose infrastructure is independent of yours.
Internal communications deserve their own plan. SMS broadcast services — used by Australian councils for emergency messaging — provide one path. Recorded phone-tree cascades documented in the secondary store provide another. Without those, you will find executives communicating by reply-all to a personal distribution list, which is fine for 20 people and catastrophic for 2,000. Have the structure ready before you need it.
Recovery models: hot, warm, and cold standby in local terms
Australian organisations choose between recovery models based on recovery time objective, budget, and the consequences of going dark. Hot standby keeps parallel infrastructure always running, ready to take over in minutes — expensive, but chosen by the major banks, the ASX, and large retailers. Warm standby keeps configuration and synchronised data but spins up within hours — a frequent choice for healthcare providers and mid-tier financial services. Cold standby keeps backups and runbooks but rebuilds from scratch over days — common in smaller organisations and critical infrastructure operators in regional areas such as Kalgoorlie, Mount Isa, Gladstone, Port Hedland, and Weipa.
This last point is worth pausing on. Many Australian enterprises operate sites far from capital-city infrastructure. Mining, energy, water utilities, and remote port operations rely on datacentre capacity that may be hundreds of kilometres from any secondary facility. Recovery models in those contexts often lean on cloud-based warm standbys in AWS Sydney or Microsoft Azure Australia East, paired with runbooks that ship operators to a regional office if local systems fail. The willingness to invest in that redundancy has risen since the SOCI Act made sector resilience an explicit regulatory expectation.
Cloud-first warm and cold standbys have largely displaced purely on-premises approaches in Australia because they remove the failure domain that ransomware depends on. A backup repository in immutability-enabled S3 or Azure Blob with object lock, replicated from a tenant the adversary cannot reach, gives you an eradication anchor. The principle is straightforward: the recovery system must live outside the compromise path, or it will be compromised with everything else.
When the burn-out fades and the after-action review begins, capture in the secondary document store what was used and what was not. The recovery patterns that get refined each year — paper failover notebooks, secondary bridges, hardware keys in sealed envelopes — become cheaper to maintain and faster to execute over time. Run a no-email drill this quarter, log the friction points, and shorten the next iteration.