What to do when your server is hacked: a response plan

Do not wipe it yet. The instinct after finding a compromised server is to delete everything and start again within the hour, and it is the one move that guarantees you will be compromised a second time the same way. The order that works is: contain the machine so it stops doing damage, preserve enough evidence to answer questions later, rotate every credential it ever held, rebuild rather than clean, restore data from before the intrusion, and only then reopen. Finding out how they got in is not the first step, but skipping it entirely is why people do this twice.

Two assumptions make the rest of the decisions easy. First, assume the attacker had everything the server had: every password in a config file, every API token, every SSH key, every database row. Second, assume you cannot trust anything on the machine to tell you the truth, including the output of its own commands. Work from those two and you will not talk yourself into a shortcut.

This post is written for the scale most people are at: one game server, one website, one small VDS, no security team. It is not a forensics manual. Where the honest answer is "this needs an expert", it says so.

How you find out, and which signs are real#

Most people do not find out from a monitoring alert. They find out because something behaves oddly, or because someone else tells them. The reports that are usually genuine:

• A message from your host or their upstream. Abuse notices are sent because traffic was measured, not guessed. Treat one as confirmed until proven otherwise.
• CPU pinned at 100% by a process you did not install. Cryptominers are the most common payload on small servers because they are the easiest to monetise. They often disguise themselves with a plausible name like kworkerd or [kswapd0].
• Outbound connections to addresses and ports you cannot explain, especially on a box that should only ever answer connections.
• Files changed at a time nobody was working. A modified index.php, a new .htaccess with a redirect in it, a PHP file inside an uploads directory.
• Accounts and keys that appeared. A new line in /etc/passwd, a new key in authorized_keys, a new administrator in your CMS, a new subuser on your panel account.
• Your site serving something else to search engines. Spam links visible to a crawler and invisible to you is a classic, and the first sign is usually a ranking drop or a browser warning.
• In-game: commands you did not run appearing in the console, admin rights granted to unknown players, items or currency that cannot be explained, a world edited at four in the morning.

Signs that are usually not a compromise: a spike in failed SSH logins (that is the background noise of the internet, and it is constant), a server restarting on its own (check memory first - a container hitting its limit is stopped and restarted clean, which looks dramatic and is not), and one strange log line with no other evidence around it.

The first ten minutes: contain without destroying#

The goal is to stop the machine from attacking others, spending your money, or continuing to leak data, while leaving as much as possible intact.

1. Stop the service, or take it off the network. On a panel host, press Stop. The container stops and its files stay exactly as they are, which is the best of both. On a VDS, either stop the affected service or remove the network route while keeping the machine running.
2. Do not power the machine off if you have any intention of investigating properly. A shutdown discards everything in memory, which is where a running miner, its configuration and often its command-and-control address live. For most small incidents nobody will do memory forensics, so this is a soft rule - but the shutdown also cannot be undone.
3. Do not delete anything. Not the odd file, not the strange cron job, not the user you do not recognise. You will want them later, and deleting one while the attacker still has a way in tells them you noticed.
4. Work from a machine you trust, and do not paste credentials into the compromised one. If the attacker got in through your own laptop, which is more common than people think, this is the step that stops you handing over the replacement credentials too.
5. Write down the time. Every log you read afterwards is a timeline, and the first entry in it is "noticed at 14:20 UTC". Use UTC throughout; mixed time zones ruin more investigations than missing logs do.
6. Open a ticket with your host. They can see network flows, node-level events and abuse reports that you cannot. Tell them what you have seen and ask whether they need anything preserved before you rebuild.

Preserve what you will need later#

Ten minutes of copying now saves the entire investigation. You are collecting three things: a copy of the compromised state, the logs, and a list of what changed and when.

• Take a backup of the server as it is, and lock it so rotation cannot delete it. On RE:NODE a backup slot can be locked against rotation for exactly this. Label it clearly. It is the only copy of the crime scene you will ever have, and it must never be restored onto a production server by accident.
• Download the logs before they roll. On Linux: /var/log/auth.log or /var/log/secure, the web server access and error logs, the application log, and journalctl output. On a game or app server: the console log and any panel activity log.
• List the recently changed files and keep the output:

$ find /var/www -type f -newermt "2026-09-01" -printf "%T+ %p\n" | sort | tail -50$ ls -la /tmp /dev/shm /var/tmp$ crontab -l; crontab -l -u www-data; ls -la /var/spool/cron/crontabs$ last -a | head -30$ ss -tunap$ ps auxf --sort=-%cpu | head -20

/tmp, /dev/shm and /var/tmp are worth a specific look because they are writable by everyone and are where dropped payloads usually land.

One caveat that matters: on a compromised machine, the machine's own tools can lie. A rootkit replaces ps, ls and netstat so that its own processes and files do not appear. If the numbers do not add up - CPU at 100% with nothing in ps, a disk that is full with nothing large on it - stop trying to enumerate from inside. Take the file-level copy, rebuild, and treat the investigation as something to do on the copy, offline.

Rotate every credential the server touched#

This is the step that is done half-heartedly, and it is the reason people get hit twice in a fortnight. Anything readable on that machine is now public. Rotate in this order, because the first item protects the rest:

1. Your host and panel account. Change the password from a clean machine, turn on two-factor if it is not already, sign out every other session, and delete any API key you do not recognise or need. On RE:NODE the account page lists active sessions with a sign-out button, API keys can be restricted to specific addresses, and TOTP codes come from any authenticator app with single-use recovery codes behind them. Two-factor on your panel account has the setup, including what happens if you lose both.
2. SSH keys. Generate new ones, install them, then empty authorized_keys of everything old. See SSH keys and hardening for doing this without locking yourself out.
3. Database users and passwords. Every user on every database the server could reach, not just the one in the config file it was using. The database security checklist covers what those users should be allowed to do afterwards.
4. Application secrets. Session signing keys, encryption keys, password reset salts, webhook signing secrets. Rotating a session key logs everyone out, which is the point: it also logs out the attacker.
5. Third-party tokens. Discord bot tokens and webhook URLs, Steam game-server tokens, a Cfx.re key, payment API keys, mail sending keys, deploy keys and personal access tokens on your git host. Revoke rather than change where you can, so the old value stops working immediately.
6. RCON and admin passwords, and any in-game admin list. RCON safely explains why an RCON port with a weak password is a remote shell for practical purposes.
7. Anything reused anywhere else. If a password on that server is also your email password, that is now the most urgent item on this list and you should have done it first.

While you are there, read the account's activity log and its list of subusers, teams and grants. An attacker with panel access adds a quiet second account far more often than they do anything dramatic. Subusers and least privilege covers what those grants should look like.

Rebuild rather than clean#

Cleaning a compromised system means proving a negative: that nothing anywhere on the disk will run again on the attacker's behalf. On a machine with a package manager, a web root, a cron system, several service accounts and a few thousand files you did not write, that proof is not available to you. Persistence is cheap and boring - a cron entry, a systemd unit, a .bashrc line, an SSH key, a modified plugin, an LD_PRELOAD entry, a scheduled task in your CMS - and you only have to miss one.

So: reinstall. The rule for what may come back from the old server is simple. Data can come back. Code and binaries cannot.

• Comes back: database dumps (inspected), world saves, uploaded images and documents, configuration you can read line by line.
• Does not come back: the application itself, plugins, mods, themes, anything executable, anything you cannot account for. Reinstall those from their original source at a current version.

On a panel host this is easier than it sounds, because the container is already an isolated, disposable thing. Reinstalling the server gives you a clean image, and one container per server means a compromise inside it did not become root on the machine. On a VDS you are reinstalling the operating system, which is why the first hour on a new VDS checklist is worth having written down before you need it at speed.

Inspect anything you carry over. A database can hold injected content as well as data: rows of spam HTML in a posts table, a hidden administrator, a stored script tag in a profile field. Uploads directories can hold PHP files masquerading as images. Grep the dump before you load it, and configure the new server so that nothing in an uploads directory can ever execute.

Restoring data without restoring the problem#

Now the backup question, and it is not "which is the newest".

1. Establish when it started. Use your preserved list of changed files and the logs to find the earliest evidence. It is usually earlier than you think - weeks between the break-in and the payload is normal.
2. Choose a backup from before that point. Newer backups may contain the back door, which is how people restore, get reinfected in a day, and conclude that the attacker "is still in the network".
3. This is why you keep more than one generation. A single nightly backup that overwrites itself gives you exactly one choice and it is probably contaminated. Backups that actually restore makes the case at greater length.
4. Restore into somewhere isolated first and look at it before it is serving anybody.
5. Accept the data loss between the clean backup and now, or merge selectively by hand. Merging is slow and worth it for a shop or a long-running world, and it must be done as data, not by copying files wholesale.

If the only backups you have are newer than the compromise, you are choosing between losing everything and restoring something you cannot fully trust. Rebuild clean, restore only the data you can inspect, and rotate everything again afterwards.

Finding the door#

Now the useful question. In practice it is nearly always one of these, roughly in order of frequency:

The last one deserves emphasis. Information-stealing malware on a laptop takes browser sessions, saved passwords and SSH keys in one go, and every subsequent login is genuinely yours. If you rotate everything from the same infected machine, you have handed over the new credentials as well. If nothing on the server explains the intrusion, scan the machines that administer it before you conclude it was magic.

For game servers specifically, an unofficial plugin build is the standard route, and the usual tell is a plugin downloaded from a forum post or a re-upload site rather than from the author. Keeping a modded server clean covers sourcing and verifying them.

Coming back online, and what to watch#

Bring it back deliberately:

• Patch first. Current operating system packages, current CMS and plugins, current game build. The exploit that worked once will be tried again within days by automated scanners.
• Close what does not need to be open. Every listening port should have a reason. Firewall rules that matter is the short version: default deny, then allow named things.
• Put two-factor on everything that offers it, and use keys rather than passwords for SSH.
• Turn on the logging you wished you had had. Connections, authentication, administrative actions, file changes in the web root. Logs worth keeping covers what to retain and for how long.
• Add [fail2ban](/blog/fail2ban-guide) on a VDS so that repeated failures cost the attacker something.
• Watch for a fortnight. Check the authentication log, the process list and outbound connections daily at first. Reinfection, when it happens, usually happens quickly.

Tell the people who are owed the truth. If player or customer data was on that server, say so plainly and say what you have changed; if payment or personal data was involved, there may be a legal duty to notify in your jurisdiction and that is worth ten minutes of proper advice rather than a guess. A short, factual notice costs you far less than the version where people find out later.

FAQ#

Should I just delete the server and start over?

Eventually yes, and that is the right instinct. Do two things first: take and lock a backup of the compromised state, and copy the logs off. They take minutes and they are the only way to answer "how did this happen" later. Then rebuild rather than clean.

Can I remove the malware and carry on?

You can remove the part you found. You cannot prove you found all of it, because persistence takes one line in one file among thousands. If the machine did anything you care about, reinstall it. The only case for cleaning in place is a single, well-understood, contained item such as one injected file in a static site with no server-side code.

How did they get in if my password was strong?

Strength stops guessing and nothing else. Reuse, an old leak, malware on your own computer, an out-of-date plugin or a secret committed to a repository are all routes that never touch your password's entropy. The table above is the list to work through.

Is my host responsible for this?

They are responsible for the platform: the hardware, the network, the isolation between customers, and the panel's own security. You are responsible for what runs inside your server, the software versions, the mods and the credentials. A compromise of your application or a leaked password sits on your side of that line, but tell them anyway - they can see traffic you cannot and will tell you if others were affected.

Will restoring a backup fix it?

Only if the backup predates the break-in and you have also rotated the credentials. Restoring a contaminated backup reinstalls the back door, and restoring a clean one while the attacker still holds a valid key or password just gives them a fresh server. Do both, in that order.

How long do attackers usually sit there before doing anything?

Often weeks. Automated compromises frequently install nothing but a quiet foothold at first and come back later, which is exactly why the date of the earliest evidence matters more than the date you noticed, and why a single generation of backups is not enough.