Running our own infrastructure

The tools we work with every day, from our accounts and project tracking to our source code and local AI models, run on infrastructure we built ourselves, on hardware we own. What we build for clients runs elsewhere, on infrastructure chosen for each of them. Here's how ours is made, and what running it has taught us.

Why we run our own

We help organizations look after the systems they already have. Most of what we'd tell them, we learned here first, usually by getting it wrong once.

Written as code, from the machines up

A hypervisor runs the virtual machines and a separate service host runs everything else in containers. Terraform declares the machines, and about forty Ansible roles take a fresh install to a working server. Rebuilding a machine means running the code again, not remembering how it was done.

One command keeps it current

It updates every machine, applies the code, checks that names still resolve and services still answer, and ends by saying what changed. Smaller updates to the software arrive on their own once the tests pass. A major version waits for us to decide.

Private by default

More than fifteen services sit behind one front door that gives each a valid certificate, and none of them can be reached from the internet. The network is split into segments, with the traffic between them limited at the gateway and again on the machines themselves. Getting in from outside means one encrypted tunnel.

Watched from outside itself

Every service is checked over HTTPS, because a running machine doesn't mean a working service. Every nightly job reports in when it finishes, so a job that quietly stops is noticed as surely as one that fails. And because everything that watches the rest runs on one machine, a second machine watches that one.

Four layers of backup

Snapshots every fifteen minutes. Every database and every service's data copied each night. An image of every virtual machine each night. And an encrypted copy off site every day. Restores are practiced from a written drill, because a backup nobody has restored is only a hope.

Local AI for what's private

A graphics card given to its own virtual machine runs language and image models on a segment of its own, and nothing sent to them leaves our hardware. That's where anything private goes. For work that isn't private, we use the leading hosted models as well.

What it's taught us

Each of these happened, and each one changed how it's built.

A routine kernel update took the service host off the network for over an hour. Everything that would have told us was running on that same machine. Now a second machine watches it, and it stays on the last kernel that worked until a newer one has proven itself.

One day Git over SSH stopped working, though nothing had changed. The program on disk was fine. The copy held in memory had two flipped bits. Clearing the cache fixed it, and the lesson stuck: check the copy that's actually running.

Our project tracker kept crashing, and the first crash pointed at the wrong thing. Turning that off didn't stop it. Watching closely did: eleven more crashes, all at one memory address, from unrelated code, which put the fault in the runtime underneath. An upgrade of that runtime alone ran clean for the three weeks we watched it.

Snapshots of backups that had already been pruned once held on to 1.44 TB of disk. Now the sources are snapshotted, and the backups made from them aren't.

A version tag moved in a registry, which would have quietly upgraded a service with a database behind it. Every container is now pinned to its exact build, and a change of build arrives as a change we can see and test.

For your systems

If your systems need the same care, that's our production and infrastructure work, and a security review is a good place to start.