What running a hosting platform for 500+ customers taught us

The short version: automate the way servers come back before you automate the way they are born, treat isolation between tenants as the product rather than a feature, assume every manual step will turn into a support ticket, build roles and permissions before your second customer, page a human only for things a human must fix, and sell backups as something the customer can see and press. Almost everything else we learned running RE:NODE is a consequence of those six.

The platform grew to 500+ clients and 1,200+ servers across game, VDS, web, database and storage hosting, with an average deploy time under five minutes. That is not a huge fleet by cloud standards. It is exactly the size where the shortcuts you took at fifty servers stop being cheap and start sending you email.

What follows is principles and opinions, not war stories. Where an example helps we have made it obviously hypothetical. If you run servers for other people - a game community, a small agency, a hosting side business - most of this applies to you at a tenth of the scale.

Automate recovery before you automate creation#

Every hosting project starts with the fun part: a button that creates a server. Payment clears, a container appears, a port is allocated, a password is generated, the customer gets an email. It is satisfying to build and it demos well.

It is also the least important automation on the platform. Creation happens once per server. Recovery happens every time a process dies, a node reboots, a disk fills or an update goes wrong, which over the life of a server is many times. If creation is manual, you lose a few minutes per order. If recovery is manual, you lose your nights.

The rule we ended up with: before a new kind of server is allowed to be sold, the platform must be able to answer three questions about it without a human.

1. Is it supposed to be running?
2. Is it actually running?
3. If those two disagree, what is the safe thing to do about it?

That third question is where the design lives. "Restart it" is the obvious answer and often the wrong one. Say a game server crashes on start because a mod is broken. Restarting it forever just burns CPU on a shared node and fills the log. So the recovery logic needs memory: it has to know how many times it has already tried, and when to stop trying and tell someone.

On RE:NODE that became a watcher that polls every two minutes for uptime that went backwards or a server that went offline. Restarts the customer asked for are not counted. Three unrequested restarts in an hour put a warning on the server page and open a ticket automatically; six suspend the server. The numbers matter less than the shape: automatic recovery, then automatic escalation, then a hard stop. If you want the customer-facing side of that, we wrote it up in why your game server keeps restarting.

The other thing recovery forces on you is idempotency. A recovery action will be retried - by the watcher, by a person, by a second copy of the watcher you forgot was running. "Start this server" must be safe to run twice. "Allocate a port" must notice the port is already allocated. Every action we wrote that was not idempotent eventually ran twice, and the result was always worse than if it had not run at all.

Desired state, not scripts#

The second principle follows from the first. Early on, provisioning was a sequence: create the container, then attach storage, then allocate ports, then write the config. A sequence is easy to write and terrible to recover, because when step three fails you are left with a server that is half built, and the script does not know which half.

The model we would use from day one now is desired state and reconciliation. The database says what should exist: this server, this much memory, these ports, this image, running or stopped. A loop compares that with what actually exists on the node and closes the gap one safe step at a time. Creation is just reconciliation from nothing. Recovery is reconciliation from something broken. Plan changes, port changes and migrations are the same loop with a different target.

# The whole idea, minus the hard parts.def reconcile(server):    want = desired_state(server)       # from the database    have = observed_state(server)      # from the node    for step in plan(have, want):      # smallest safe steps first        if not step.apply():           # every step is idempotent            record_failure(server, step)            return                     # retry on the next pass    mark_converged(server)

None of this is new - it is how Kubernetes, Terraform and most configuration management think. The lesson is not the pattern, it is how early you need it. We would not wait for the first half-built server. The cost of retrofitting desired state onto a script-shaped system is much higher than the cost of starting with it, because by then every edge case has been handled by a special branch in a script that nobody wants to touch.

Isolation between tenants is the product#

Customers do not buy a container. They buy the assurance that their server behaves the same whether the neighbour is idle or running a badly written plugin in a loop. If you cannot give them that, nothing else on the feature list matters.

There are three resources where isolation decides whether the platform feels fair: CPU, memory and disk. The opinions we hold firmly about each:

On RE:NODE each server is its own container, CPU is a hard throttle to the share bought, and at the memory limit the kernel stops the container and it restarts clean rather than swapping - which is the opposite of the upstream Pterodactyl default. That last one surprises people. A server that is killed and restarts looks worse in the moment than a server that slowly swaps. In practice the swapping server takes the disk I/O of everyone else on the node with it, and the person who suffers most is a customer who did nothing wrong. We covered the mechanics in Linux swap and the OOM killer and the shared-hardware side in shared CPU and noisy neighbours.

The uncomfortable part is that strict isolation makes the platform look less generous. A host that lets you burst to 400% CPU for free will win a benchmark against one that throttles you to the 200% you paid for. But the burst is borrowed from your neighbours, and it disappears exactly when the node is busy, which is exactly when you need it. We would rather sell a smaller number that is true at 9pm on a Friday.

Isolation also applies to everything that is not a resource. Credentials per server rather than per account. Backups scoped to the server that made them. Logs that cannot leak another tenant's output. Ports that are allocated, not guessed. Each of these is a place where a shortcut saves an afternoon and creates a class of bug where one customer sees another customer's data. That is the one class of bug a hosting platform does not recover from reputationally.

Every manual step becomes a support ticket#

This is the principle we would put on the wall. Any step in the customer's journey that needs a human on our side will, at scale, be a ticket. Not sometimes. Every time, multiplied by the number of customers, and it will arrive at the worst moment.

The maths is simple and brutal. A step that takes five minutes of staff time and is needed by one order in ten sounds harmless. At a few hundred customers it is a part-time job, and it is a part-time job made of interruptions. Worse, the customer is waiting while it happens, so it is also a delay they notice.

The way we learned to use this: when a ticket category appears more than a handful of times, it is not a support problem, it is a missing feature. The ticket is a bug report written by someone who did not know it was a bug. Common patterns, in roughly the order a hosting platform meets them:

• "Can you open a port for me" - the customer needs a ports screen, not a person.
• "Can you restore yesterday's backup" - backups need a restore button the customer can press.
• "My friend needs access but not to billing" - you need subusers with granular permissions.
• "I uploaded the wrong file and the server will not start" - you need a file manager and a console that shows the real error.
• "Is it down or is it me" - you need a status page and per-server graphs.

Not every manual step can go. On RE:NODE game, app, web and database servers are created as soon as payment clears, usually inside a minute, but a VDS is still prepared by hand within 24 hours. That is a deliberate trade: a whole virtual machine with its own operating system has more ways to go wrong than a container from a known image, and the volume does not yet justify the automation. The point is to make that trade knowingly, write the delay on the product page so nobody is surprised, and revisit it when the numbers change.

The corollary is that the product's user interface is mostly an inventory of support tickets you no longer get. When we look at a hosting panel now, we read it as a list of things the team got tired of doing by hand.

RBAC from day one, not from customer fifty#

Access control is the feature everyone plans to add later, and later is always too late. The reason is not security theatre. It is that the first version of any panel models "a user owns a server", and the moment two people share a server - a game community with three admins, an agency and its client - that model breaks in ways that leak into every table and every permission check in the code.

What we would build on the first day, even if the user interface does not show it yet:

1. A permission is a single verb on a single kind of thing: start the server, read files, write files, see billing.
2. A role is a named set of permissions.
3. A team is a set of people.
4. A grant joins a team to a server with a role.

That is the model RE:NODE ended up with: roles hold permissions, teams hold people, grants join teams to servers, with invites by link, join code or email, time-boxed access and a per-server activity log. It looks over-engineered for one customer with one server. It is the only shape we have seen survive contact with real communities, where the person who pays is not the person who restarts, and the person who restarts should not be able to delete the world.

Two details matter more than they look. Time-boxed access, because the most common access mistake is not giving someone too much, it is forgetting to take it away. And the activity log, because when something goes wrong the first question is always "who did that", and without a log the answer is an argument. The customer-side advice is in subusers and least privilege.

The same rule applies to staff. Support should be able to see a server's state without being able to read its files, and nobody on the team should need the root password of a node to do their daily job. If staff access is all-or-nothing, it will be all, for everyone, forever.

Monitoring that pages for the right things#

A monitoring system that pages you for everything trains you to ignore it. A monitoring system that pages you for nothing is a dashboard. The useful version sits in between and is mostly a decision about who should be woken up, and for what.

The rule we settled on: a page goes to a human only when three things are true. Customers are affected or about to be, the platform cannot fix it on its own, and waiting until morning makes it worse. Everything else is a ticket, a graph or a log line.

That last row is the one people argue with. A customer's server at 100% of its CPU share is slow, not broken, and the platform should not page anyone for it or suspend it for it. It is doing exactly what the throttle says. Alerting on it creates noise that hides the signals that do matter.

The second rule: alert on the symptom and the trend, not on the threshold alone. Say a node's disk is at 85% at 3am. That alone is not urgent. If it was at 60% at midnight, it is very urgent. A rate-of-change check catches the runaway log or the backup that forgot to prune long before a flat threshold would, and it does not fire on a node that has sat at 85% quietly for a month.

The third: every page has a runbook link, even a short one. A half-asleep responder does not reason well, and a page that says "node disk filling" plus three lines on where to look first is worth more than a beautiful dashboard. We wrote the general version of this in monitoring that tells you something.

A page that does not need a human to act is not an alert. It is a habit you are building of ignoring alerts.

Backups are a product feature, not an ops job#

Most platforms treat backups as something operations does quietly in the background. We think that is wrong for a hosting company, because the backup the customer cannot see is the backup the customer does not trust, and the backup the customer cannot restore themselves becomes a ticket at the worst possible moment.

So there are really two backup systems, with different jobs:

• Platform backups protect the business: nightly, copied to separate hardware, verified, pruned on a schedule. Customers never touch them. They exist for the day a node dies.
• Customer backups protect the customer from themselves: slots on every plan, taken on demand or on a schedule, restored with a button, downloadable, lockable against rotation, and stored off the machine they protect.

The customer-facing one is the one that changes support volume. Most data loss on a hosting platform is not hardware failure. It is a customer installing a plugin that eats the world, or deleting the wrong folder, or upgrading a game version that cannot read the old save. A restore button that the customer presses at midnight without asking anyone is worth more than any amount of redundancy on our side.

Two design decisions worth repeating. Locking: the ability to pin one backup so rotation never deletes it, because the backup you want is often the one from before the problem started, which is precisely the one rotation is about to remove. And honest deletion: on RE:NODE, deleting a server deletes its backups, locked ones included. That surprises some people, but a backup that outlives the server it belongs to is data you are holding without a reason, and it should be stated plainly rather than discovered.

The principle underneath is the house line: a backup nobody has restored is a hypothesis. Verification has to be automatic, because a person checking restores by hand will stop doing it the week they get busy. Backups that actually restore and testing a restore before you need it go into the mechanics.

Incident response is mostly preparation#

The incident itself is the smallest part of incident response. The parts that decide how it goes happen weeks before: whether you can tell which customers are affected, whether you can tell them quickly, and whether you can find out afterwards what actually happened.

What we would insist on in any platform at this size:

• A way to go from "node X is unhealthy" to "these customers are affected" in one query. If that takes a spreadsheet, the status update will be late and vague.
• A status page that is updated by a person during an incident and by the system the rest of the time, hosted somewhere that does not share fate with the thing it reports on.
• Every automated action written to an event log with what, when, why and which component did it. When the recovery loop restarts something, that restart should be explainable a week later.
• A short written review after anything customers noticed, focused on what the system allowed to happen, not on who pressed what.

The last one is about culture more than tooling. If the review hunts for a person, people stop reporting near misses, and near misses are the only free lessons you get.

What we would do differently#

If the platform were started again tomorrow, knowing what we know now, these are the changes we would push for. None of them is dramatic. All of them are cheaper on day one than on day one thousand.

Desired state from the first commit

We covered this above, but it is the biggest one. Every provisioning step written as a script is a future migration project. We would have one reconciliation loop from the start and make creation, change and recovery the same code path.

One place for limits and quotas

Limits end up scattered: memory in the container definition, disk in a quota tool, backup slots in the billing system, ports in the allocation table. Each is correct on its own and together they drift. We would keep every limit for a plan in one record that every component reads, so changing a plan is one write, not five.

The event log as a first-class product

We would build the audit and event log before the dashboard. It is what support reads, what the customer's activity view shows, what incident reviews depend on and what you need for any compliance question. Retrofitting it means guessing at history you did not record.

Location as data, not as an assumption

Even a platform that runs in one place should not assume it in the code. We would model location as a property of a node and a server from the start, so that adding a second one is a configuration change rather than a rewrite. Not because a second one is planned, but because hard-coded assumptions are the expensive kind.

Test the recovery path on a schedule

Recovery code is the code that runs least and matters most, which is a bad combination. We would deliberately break a test server on a schedule - stop its process, fill its disk, corrupt its config - and check that the platform notices and does the right thing, the same way restores should be tested on a schedule. If recovery only runs during real failures, it is tested only during real failures.

What we would keep exactly as it is

Strict resource limits. A watcher that escalates instead of restarting forever. Customer-pressable backups. Granular roles. A panel that shows the real console output instead of a friendly summary. Each of these made the platform feel less generous on day one and saved far more than it cost.

A lot of this comes down to one habit: designing for the failure first. It holds even when AI coding agents write much of the code: an agent can write the creation path in an afternoon, and the recovery path is still the part that needs a human to think hard.

FAQ#

What is the hardest part of running a multi-tenant hosting platform?

Fairness under load. Creating servers is easy; making sure one tenant's bad day does not become everyone's bad day takes hard limits on CPU, memory and disk, and the willingness to look less generous than hosts that let customers borrow from their neighbours.

Should a small host automate provisioning or recovery first?

Recovery. Provisioning runs once per server and a slow manual version costs minutes. Recovery runs every time something breaks, usually at night, and a manual version costs your sleep and your customers' trust. Make sure the platform can detect, retry and escalate before you make the order button instant.

Is RBAC worth it for a platform with only a few customers?

Yes, at least in the data model. Retrofitting shared access onto a "one user owns one server" design touches every permission check in the code. You can hide roles and teams in the interface until customers ask for them, but the tables should exist from the start.

How many alerts should page a human?

Very few. Page only when customers are affected or about to be, the system cannot fix it alone, and waiting makes it worse. Node down and a disk filling fast qualify. A failed backup is a morning ticket. A single server at full CPU is not an alert at all.

Why stop a server at its memory limit instead of letting it swap?

Because swap on shared hardware turns one tenant's memory problem into everyone's disk problem. A clean restart is visible and fixable by the customer; a node quietly thrashing affects people who did nothing wrong and is much harder to diagnose.