Persistence on Linux servers and how to remove it
Linux servers often hold the data, identities and execution paths that keep Australian organisations running. After an attacker gains a foothold, the enduring problem is usually access rather than the initial exploit. The challenge of removing persistence in Linux-based servers starts with the fact that unauthorised changes can look like normal administration: an SSH key, scheduled job or service account may be entirely plausible.
Persistence also exists outside the filesystem. Valid credentials, active sessions, cloud roles and stolen automation tokens can restore access without installing malware. A reboot may remove an in-memory implant, yet it does nothing to a malicious cron entry, a modified binary or an identity stored in an external directory. Cleanup therefore needs a system-wide view of how the host starts, communicates and authenticates.
Australian environments add specific operational pressures. Sydney and Melbourne are major centres for finance, government and cloud workloads, while Brisbane and Perth support mining, energy, logistics and healthcare operations. Many organisations align with the Australian Signals Directorate and ACSC guidance, including the Essential Eight, while financial institutions also face APRA CPS 234 and the Privacy Act. Local systems often combine modern cloud services with legacy appliances, making one-size-fits-all remediation risky.
Coordinated response matters because endpoint, identity, network and cloud teams may hold different evidence. The Australian market often combines internal operations with managed-service providers in follow-the-sun models, sometimes supported from Singapore and India, adding handoffs across time zones. CARM Security, developed by Exclusive Networks, provides a relevant ecosystem for this stage by integrating multiple security vendors around post-breach attack remediation and mitigation.
Why Linux persistence is hard to see
Linux gives administrators many legitimate ways to make software run automatically. Attackers use those same controls to blend in, and a clean file scan cannot expose every authorised path. Their objective may be a hidden account, altered startup script, malicious library or a command scheduled to run under an existing service.
The persistence inventory should distinguish access from execution and configuration drift. Evidence from a healthy peer, package records and change management can separate hostile artefacts from approved automation.
| Persistence layer | Typical examples | Removal challenge |
|---|---|---|
| Credential persistence | Unauthorised SSH keys, password changes, sudoers entries and stolen tokens | Revoke access without breaking valid administration |
| Scheduled execution | Cron, at jobs, systemd timers and shell profiles | Identify inherited jobs and time-triggered payloads |
| Startup control | Init scripts, service units, udev rules and PAM modules | Restore the trusted boot sequence and permissions |
| Code and runtime | Modified binaries, preload libraries, kernel modules and injected processes | Establish whether the kernel and filesystem are trustworthy |
| Cloud and workload | IAM roles, metadata access, CI/CD secrets and container manifests | Remove external footholds as well as host artefacts |
These categories can overlap. For example, a stolen key may be copied into authorized_keys and then used to create a scheduled task. Each link in that chain needs removal, including credentials that still work from another machine.
Map every persistence pathway
Start with an evidence-led inventory of the host, its identities and its workloads. Collect process, service, timer, cron, kernel-module, package and file-integrity data; list authorised SSH keys, sudoers rules, PAM changes and recent account or privilege changes. Record the expected services and configuration for that server role, ideally from a known-good image or peer in the same environment.
Compare the inventory with authentication and network telemetry. Sign-ins from unusual locations, new token use and unusual outbound connections may reveal persistence that leaves few local traces. Current host findings should also be compared with the wider context from AppOssum security intelligence, but external intelligence should validate—not replace—local evidence.
Extend the search to the cloud control plane. Audit instance roles, security groups, metadata settings, CI/CD variables, deployment keys, container images and Kubernetes workloads. An attacker who compromises a pipeline can restore access after the server is rebuilt, so host eradication without cloud remediation leaves the same pathway open. Include managed-service accounts and vendor access in this pass, then record each owner and approval.
Contain access while preserving evidence
Isolate the affected workload before deleting anything, while balancing availability with containment. Preserve volatile evidence such as process state, memory, active network sessions and recent authentication events where feasible. Capture a forensic image or point-in-time snapshot, record system time and collect relevant logs from the host, SIEM, identity provider, firewall and cloud audit trail. Where service continuity matters, use a controlled network block rather than an abrupt shutdown.
Revoke compromised access from a trusted management device. Disable or reset affected accounts, remove attacker-created keys, invalidate sessions and tokens, rotate secrets and service credentials, and close suspect egress paths. Prioritise privileged accounts, CI/CD credentials and reusable API keys because each can support persistence beyond the original server. Cloud and identity teams should work from independent control planes to avoid trusting changes made by the compromised host.
Containment should be proportionate to the evidence and the operating environment. A suspected miner in a Perth energy network requires different handling from an isolated web server in Melbourne, and production or medical systems may need staged isolation during approved maintenance windows. Track containment decisions in the incident log, preserve a chain of custody and maintain heightened monitoring for renewed access.
Remove mechanisms and rebuild trust
Begin eradication from the verified inventory, not from a generic cleanup script. Disable malicious timers and services, restore approved cron jobs and startup files, remove unauthorised keys and modules, repair package and file integrity, and correct permissions or ownership. Compare the result with the known-good baseline, then restart only what the evidence and change plan require. A second scan is useful, but absence of a known indicator is weaker than a positive match to trusted configuration.
Rebuild rather than repair when the integrity boundary is broken. Kernel-level implants, a hidden rootkit, unexplained firmware changes or widespread tampering make confidence in the running operating system unrealistic. Replace the instance from a trusted image, rotate every credential and token that touched it, patch the initial access path, and preserve the old disk as evidence. Rebuilding alone is not enough if the attacker can return through an unchanged identity or deployment pipeline.
Validation must use an independent perspective. Check endpoint alerts, identity events, service creation, outbound traffic and file integrity after cleanup; hunt for related indicators across other servers. The integrated view offered by the CARM Security ecosystem can support this identify-contain-respond-remediate sequence, while local teams provide the operational context needed to distinguish hostile persistence from sanctioned configuration. Recovery is credible only when access, execution and trust have all been re-established.
A practical verification checklist
A repeatable response standard reduces reliance on individual memory under pressure.
- Mark every affected host, account, service and credential, then define the trust boundary around each.
- Capture volatile and durable evidence before rebooting, restarting or deleting suspicious files.
- Compare startup paths, scheduled jobs and administrative controls with a known-good baseline.
- Revoke sessions and tokens, rotate exposed secrets and block reinfection routes from a clean device.
- Remove or restore malicious artefacts, then validate package, configuration and file integrity.
- Rebuild systems where operating-system integrity cannot be established, including build and deployment hosts.
- Monitor identity, endpoint and network telemetry after recovery and retain a time-limited watch for recurrence.
Document what was malicious, how it entered, which systems were affected and what evidence supports each removal decision. This record gives regulators, insurers, law-enforcement partners and internal assurance teams a defensible account, particularly where Australian privacy obligations or sectoral reporting duties apply. It also exposes recurring gaps in hardening, identity governance and supplier access.
The practical takeaway is to treat a Linux server as untrusted until its identities, startup paths and artefacts have been independently verified; only then does persistence removal close the breach instead of merely resetting the countdown.