Rebuilding a Compromised Kubernetes Cluster From Scratch
A Kubernetes breach can leave an organisation with an uncomfortable question: can the existing cluster still be trusted? Once an attacker has obtained privileged access, persistence may exist in admission controllers, service accounts, container images, nodes, secrets, or cloud integrations. Removing one suspicious pod rarely proves that the environment is clean.
A rebuild from trusted foundations is often safer than attempting to repair every altered component in place. The process must preserve evidence, limit business disruption, restore workloads from verified sources, and establish stronger controls before production traffic returns. For Australian organisations, the approach also needs to account for the Privacy Act, the Notifiable Data Breaches scheme, sector obligations such as APRA CPS 234, and data residency expectations.
Decide Whether the Cluster Can Still Be Trusted
The first decision is whether the current control plane and worker nodes remain reliable sources of truth. Indicators for a full rebuild include unauthorised cluster-admin activity, modified control-plane components, unknown admission webhooks, compromised cloud credentials, suspicious DaemonSets, altered audit settings, and evidence that an attacker moved between nodes.
A cluster may also warrant replacement when the scope cannot be established with confidence. If logs were disabled, credentials were shared, or an attacker had prolonged administrative access, a clean-looking environment may still contain hidden persistence. Treating uncertainty as a security finding is more defensible than accepting a narrow remediation result based on incomplete telemetry.
A rebuild does not mean destroying evidence immediately. Isolate the environment from public ingress and non-essential outbound connections, preserve relevant snapshots, and record the time, systems, accounts, and decisions involved. The incident response team should define the point at which the old cluster becomes a forensic artefact rather than a recovery platform.
Contain the Incident Before Destruction
Containment should begin outside Kubernetes. Revoke or rotate cloud access keys, CI/CD tokens, registry credentials, database passwords, TLS private keys, signing keys, and service-account tokens that may have been exposed. Check identity-provider sessions and privileged accounts, because a clean cluster is still vulnerable if the attacker retains access to the surrounding platform.
Use network controls to separate the suspected environment from production databases, corporate identity services, backup systems, and management networks. Keep a controlled path for forensic collection and approved administration. In a Sydney or Melbourne cloud region, this may involve security groups, private endpoints, transit gateways, and provider support processes rather than simply deleting a virtual network.
Capture useful evidence before terminating nodes. Preserve control-plane audit logs, cloud activity records, container runtime logs, ingress records, identity events, DNS queries, and images or manifests associated with suspicious workloads. Record hashes and collection times, and store copies in an access-controlled location that the compromised cluster cannot modify.
Incident findings must eventually be understandable to executives, regulators, insurers, and operational teams. A clear incident reporting guide helps translate technical observations into business impact, decisions, timelines, and residual risk without weakening the underlying evidence.
Build a Trusted Recovery Foundation
Create the replacement environment through a separate, hardened administrative path. Use a clean management workstation, newly authenticated administrator accounts, phishing-resistant multifactor authentication, and freshly issued credentials. Avoid copying kubeconfig files, shell history, scripts, Terraform state, container images, or secrets from the compromised environment until each item has been reviewed.
Pin approved Kubernetes versions and node images, and obtain them from trusted repositories with signature or checksum verification. Apply a defined baseline before workloads arrive: private control-plane access where practical, restricted API exposure, encrypted secrets, audit logging, network policies, admission controls, and separate node pools for different trust zones.
Infrastructure as code should describe the cluster, identity bindings, policies, logging, and dependencies in a version-controlled repository. That repository must itself have protected branches, reviewed changes, strong authentication, and a recovery path independent of the old environment. A build that is repeatable is easier to test and less likely to reintroduce undocumented weaknesses.
Australian organisations should map the design to relevant obligations and assurance requirements. The Australian Cyber Security Centre’s guidance and the Essential Eight provide useful reference points, while regulated financial entities may need to demonstrate resilience under APRA expectations. A local healthcare provider, university, or government supplier may also need clear evidence about where logs, backups, and personal information are stored.
Restore Workloads From Verified Sources
Treat application recovery as a supply-chain exercise. Rebuild container images from known source commits where possible, scan dependencies and operating-system packages, verify image signatures, and compare release artefacts with records from before the compromise. Do not assume an image is safe because it is present in a trusted registry; registry credentials or build runners may have been abused.
Restore manifests selectively rather than importing the entire old cluster. Review Deployments, StatefulSets, Jobs, CronJobs, ConfigMaps, Roles, RoleBindings, ServiceAccounts, Ingress objects, custom resources, webhooks, and storage definitions. Pay particular attention to privileged containers, hostPath mounts, host networking, wildcard permissions, startup scripts, and encoded values hidden in configuration.
Persistent data requires a separate trust decision. Validate backup provenance, scan restored files, check database users and grants, and examine scheduled tasks or stored procedures for manipulation. Recovery points should be compared with incident timelines so that a backup created after attacker access is not mistaken for a clean copy.
| Recovery area | Verification before use | Safer rebuild practice |
|---|---|---|
| Container images | Source commit, digest, signature, dependency scan | Rebuild in a clean pipeline and deploy by immutable digest |
| Kubernetes manifests | Privileges, webhooks, mounts, service accounts | Recreate from reviewed definitions rather than exporting blindly |
| Secrets | Exposure history and access logs | Reissue from a clean secrets platform with narrow permissions |
| Persistent data | Backup age, integrity, account changes | Restore a tested point and validate application behaviour |
| CI/CD systems | Runner integrity, tokens, pipeline edits | Recreate runners and rotate every deployment credential |
| External services | Identity bindings, callbacks, firewall rules | Reauthorise integrations individually and monitor first use |
Recreate Identity, Secrets, and Delivery Pipelines
Identity is often the most important boundary in a Kubernetes rebuild. Establish separate administrator, operator, developer, and workload roles. Remove broad cluster-admin access, avoid long-lived static credentials, and use short-lived tokens or federated workload identities where the cloud platform supports them.
Secrets should be generated again, not copied from old manifests or backups without review. Rotate database credentials, message-queue passwords, API keys, certificates, encryption keys, and registry tokens. Sequence rotations carefully: applications may need temporary dual credentials, but the old values must have a defined expiry and verified revocation.
Rebuild the delivery pipeline as part of the environment. Inspect source repositories for unauthorised commits, pipeline modifications, dependency confusion, exposed variables, and compromised runners. Use isolated build workers, protected environments, mandatory reviews, software bills of materials, and promotion based on signed artefacts. A deployment should be traceable from source revision to image digest to running workload.
For companies operating across Perth, Brisbane, Sydney, or Melbourne, outsourced platform operations are common, particularly where a small security team supports a large Kubernetes estate. Contracts should define who can access production, how emergency changes are recorded, where telemetry is retained, and how the provider participates in incident response. Responsibility can be shared, but accountability for access and recovery decisions cannot be vague.
Validate the Rebuilt Cluster Before Cutover
Validation should proceed in stages. Start with control-plane health, node integrity, admission behaviour, identity controls, logging, backup jobs, network segmentation, and policy enforcement. Then deploy a small representative workload and test its normal dependencies, failure modes, scaling behaviour, and rollback path.
Run security checks against the rebuilt environment: enumerate permissions, test network policies, inspect exposed services, verify pod-security settings, review cloud roles, and confirm that audit events reach an independent monitoring platform. Check that alerts fire for suspicious actions such as privilege escalation, creation of privileged pods, secret access spikes, and changes to admission controls.
Conduct a controlled cutover rather than switching every service at once. Use a blue-green or staged migration where possible, directing a small percentage of traffic to the replacement environment before increasing the share. Watch error rates, authentication events, outbound connections, DNS activity, resource consumption, and unexpected image pulls.
The old cluster should remain isolated until the response team agrees that evidence collection is complete and dependent systems are safe. When it is decommissioned, remove credentials, revoke certificates, destroy storage securely, and retain required forensic material under documented access controls. Deletion is an operational step, not proof that the incident has ended.
Monitor for Recurrence and Prove Recovery
A rebuilt cluster needs heightened monitoring because the attacker may still target adjacent systems. Correlate Kubernetes audit events with cloud control-plane logs, identity-provider activity, endpoint telemetry, registry events, DNS records, and firewall data. Independent log storage is essential; an attacker who can alter the cluster should not be able to erase the only record of their actions.
Define concrete recovery success criteria. These might include zero unknown privileged identities, verified image provenance, complete secret rotation, restored backup testing, expected application behaviour, confirmed regulatory notifications where required, and a period without indicators of compromise. Document who accepted each risk and which assumptions remain unverified.
Use the incident to improve future rebuild speed. Maintain an isolated recovery account, tested infrastructure templates, current dependency inventories, clean golden images, emergency contact details, and rehearsed runbooks. Tabletop exercises should include technical recovery and executive decisions, including service prioritisation, customer communications, legal review, and interaction with insurers or the Australian Cyber Security Centre.
CARM Security’s role in this kind of response is particularly relevant when multiple security technologies and vendors must work together. Detection, containment, investigation, remediation, and reporting are more effective when they form a coordinated process instead of separate tool-specific activities. The target is a defensible recovery record, not simply a cluster that happens to start.
Make the New Environment Harder to Compromise
A clean rebuild is an opportunity to reduce blast radius. Separate development, staging, and production; isolate sensitive workloads with namespaces and node pools; enforce least privilege; restrict egress; and make administrative access private and strongly authenticated. Use policy as code so that unsafe settings are rejected before deployment rather than discovered after exposure.
Prioritise controls that reveal misuse quickly. Centralised audit logs, runtime detection, image provenance, immutable deployment references, secrets-management integration, and alerts for unusual service-account behaviour provide useful visibility. Regularly test whether those controls still work after upgrades, migrations, and changes to cloud networking.
Recovery plans should reflect the business environment. A retailer preparing for major sales periods, a mining company operating remote sites, and a health service holding sensitive records will have different recovery priorities and tolerances. Australian time zones, public holidays, local support coverage, and dependence on cloud regions in Sydney or Melbourne should be explicit in the runbook rather than left to emergency judgement.
The key principle is simple: when trust in a Kubernetes cluster has been lost, preserve evidence, replace the compromised foundation, restore only verified components, and validate every identity and dependency. The reader should remember that a successful rebuild is measured by restored trust and demonstrable control, not by how quickly the first pod becomes ready.