Before I rewrote anything, I measured it

Deployed at Home: my path toward Forward Deployed Engineer. Part 1 of 4, discovery: the numbers that started it.

A forward deployed engineer works inside a customer’s operation. The job is to find the real problem behind the request, build against the customer’s actual environment, and stay with it until it runs in production. I want to do that work, and this series is how I’m practising it.

I didn’t have a customer to embed with, so I used the closest one I had: the back office for a small guesthouse. On this project I was the customer, the engineer and the person who gets paged. Two AI agents wrote most of the code. The four parts follow four habits I think the role depends on: turning a complaint into a question you can measure, making a call and writing down why, turning lessons from the field into tools, and working between people who can’t see each other’s screens.

A month ago I had a working back office: houses and rooms, guests, reservations with the full stay lifecycle, payments, pricing, Bulgarian holidays, a calendar and a dashboard. It was a .NET 10 API orchestrated with Aspire, a Next.js front end, Keycloak for login and PostgreSQL underneath.

It worked. Then I tried to work out what server it needed, and the answer was more RAM than I expected for an app that a handful of owners use.

The complaint, in the form a customer would bring it, was “it uses too much RAM”. This part is about turning that into something I could measure, because everything after it depends on the answer.

What I was running

The production topology that aspire publish generated was seven containers on one box:

That is one JVM, three .NET processes, one Node process and PostgreSQL. None of them is large on its own. Together they add up, and none of them had a memory limit in the generated Compose file. Keycloak’s JVM sizes its heap as a percentage of whatever memory it can see, so without a limit it sees the whole host.

Turning the complaint into a question

The tempting move was to pick a plan by feel, or to decide the stack was “too heavy” and rewrite it. I didn’t trust either. A rewrite costs weeks, and “too heavy” can’t be measured.

So I asked a narrower question: for about 100 connected owners, which Hetzner plan is the cheapest one this stack actually fits on, and how do I prove it?

The first research pass worked out what “100 clients” means for owner-only software: 100 browser tabs open on a dashboard or calendar, with a click every 45 seconds or so, plus silent token renewal every five minutes. That comes to roughly 1 to 3 requests per second on average and 10 to 15 at peak, which is about a third to half of one vCPU.

At that load the CPU is mostly idle. Each idle keep-alive connection costs kilobytes, while each runtime costs hundreds of megabytes before it serves a request, so memory was the metric to watch.

Step 1: price every plan

I pulled the prices from Hetzner’s own price feed on 2026-09-30 and listed every shared and dedicated x86 plan in the EU locations:

The ARM line (CAX) and the cost-optimized CX line were marked “currently unavailable” on Hetzner’s site that day, in English and in German, so ARM was off the table whatever it might have saved.

On paper, the per-container estimates came to about 3.3 GB of memory limits plus 0.7 GB for the host and a reverse proxy. That is exactly 4 GB, which is CPX22’s whole machine. The research recommended CPX32 at €35.49 as the safe default and CPX22 as a knife-edge option.

Estimates are not measurements, though, and I didn’t want to pay for a month of a plan to find out.

Step 2: turn the laptop into the server

The trick that made this cheap is that Podman on macOS runs containers inside a Linux VM, and you can resize that VM.

podman machine stop
podman machine set --cpus 2 --memory 4096
podman machine start

Now the Mac behaves like a 2 vCPU, 4 GB box for memory purposes: one shared memory pool, one kernel, one OOM killer for every container, and no swap. The Podman man page says --cpus and --memory only work with QEMU. The research read the Podman 5.8.4 source and found that the man page is out of date: the Apple hypervisor provider rebuilds the VM from those values on every start. Images and volumes survive the resize.

The method has one deliberate asymmetry. If a run on the shrunken VM fails, the plan is too small. If it passes, that does not prove the plan is enough, because an M5 Pro performance core is not a shared EPYC vCPU with noisy neighbours. The RAM figure transfers to the server; the CPU figure only works as a canary.

Two details mattered more than I expected.

First, the test has to run against the containerized production topology. In Aspire’s dev mode the API and the Next.js app run as Mac processes, so they have no cgroup to read and no limit to hit, and the web tier is the Next.js dev server rather than the production build.

Second, I read the kernel’s own numbers instead of sampling podman stats. Cgroup v2 keeps memory.peak for every container, so the true peak is one file read after the run. Sampling every few seconds misses spikes.

The metrics I decided on were the ones that survive the move to a real VPS: peak cgroup memory per container, CPU-seconds per thousand requests, the PostgreSQL connection peak and OOM kills. I treated local p95 latency as a smoke alarm only, because the request crosses a userspace network forwarder on the Mac that a VPS doesn’t have.

Step 3: run it

Getting to a clean run took longer than the run itself. The first real start of the containerized stack found four defects that 19 passing tests and five review rounds had missed. One of them: the seeder wrote its fixture file into the same directory Keycloak imports realms from, Keycloak tried to parse the fixture as a realm on the next restart and exited, and the symptom was a 500 from an unrelated endpoint.

The 512 MB limit I had planned for the API was too small. It was OOM-killed with exit 137. The YARP gateway crashed with SIGILL under load at both 256 MB and 1 GB, so the measured runs went straight to the API, and the gateway stayed unmeasured.

The CPX22 baseline, measured on 2026-10-01:

The API alone peaked at 741 MiB. That is the process I had budgeted at 512.

The run that said “fits” while the app was dead

The CPX12 run is the part I learned the most from.

On 1 vCPU and 2 GB the API container died. podman inspect afterwards said exit=137 oom=false: it was killed with SIGKILL, but the runtime never set the OOMKilled flag. The summariser counted OOM kills, found zero, and wrote verdict: fits.

It got worse. Because the API was dead, its cgroup directory was missing from the capture, so the peak memory sum quietly left it out: 2,132 MiB of surviving containers instead of roughly 2,873 MiB, on a 2,048 MiB machine. Over 9.6 million requests went out and 98.8% of them failed. The report still recommended the plan.

That is the dangerous direction for a sizing tool to be wrong in. The fix was simple once the bug was written down: every service that has a memory limit must have a capture directory, or the run fails, and each container’s exit state is recorded next to its cgroup numbers. The broader lesson stayed with me for the rest of the project, and Part 3 is mostly about it: a check that cannot fail tells you nothing.

What the numbers said about a rewrite

With measurements in hand, I asked the obvious next question: would a lighter stack change which plan I buy?

The research answer was no. Replacing Keycloak with ASP.NET Core Identity and dropping the gateway and dashboard would save about 0.5 GB against a tuned version of the same stack. That is €96 to €192 a year. Both versions still need a 4 GB box once the host overhead is in. The research also pointed out that this repository had already removed ASP.NET Identity once, four weeks earlier, in favour of Keycloak.

So the measurement argued against rewriting for RAM. I rewrote anyway, for a different reason, and Part 2 is about that decision and why I don’t think the numbers were wasted.

What this has to do with forward deployed work

If a client had brought me “it uses too much RAM”, the most useful first deliverable would have been this part, before any code. It defines the load in their users’ terms, lists every option with its price, and measures the one that matters on hardware that behaves like theirs. It also states plainly what the measurement can’t prove.

The CPX12 story is the part I’d most want a client to hear. My own tool told me a plan fitted while the app was dead. Reporting that as a bug, with the 98.8% failure rate in the first paragraph, is less comfortable than reporting a verdict. It’s also the only version a customer can safely act on.

If you want to repeat this

1. Define the load in your users’ terms first. “100 clients” for an owner-only app is 1 to 3 requests per second, and that tells you to watch memory.
2. Price every plan you could buy, and check availability on the day. Mine changed the answer for ARM.
3. Shrink a local VM to the plan instead of renting it. With Podman on a Mac that is podman machine set while the machine is stopped.
4. Run the production topology, with memory limits that equal swap limits so nothing borrows swap the server won’t have.
5. Read memory.peak from cgroup v2 rather than sampling.
6. Make a missing service fail the run. A dead container should never shrink the total.
7. Trust a local failure. Treat a local pass as a lower bound on RAM and nothing more on CPU.

Summary, and how this was actually built

I didn’t do this by hand, and the article so far reads as if I did. The work was done by AI agents running under a harness I set up, with me deciding what to ask and what to believe.

• Tools. Claude Code and DeepSeek Harness (DSH), the harness that runs DeepSeek V4.1 Flash. Guardrails written as code sit around the agents, built on a private agent SDK. Part 3 covers the ideas behind them.
• What the agents did. The research reports (Hetzner sizing, load testing, the rewrite question) came from a research agent. The sizing harness (resizing the VM, k6 workload, cgroup capture, summariser) was built as a plan of tasks, T1 to T5, each executed by specialised agents against a written definition of done.
• What I did. I asked the question, chose the metrics, ran the first real start of the stack, read the results, and decided what to believe. The CPX12 measurement was one I took by hand, without the driver script that would have caught the dead API. The bug report for it is mine.
• What went wrong anyway. Nineteen passing tests and five review rounds missed four defects that only a real start found. The summariser, written by agents under those gates, called a dead app “fits”. Agents write the code, the harness checks it, and a person still has to read the result.

The numbers in this part:

1. “100 clients” for an owner-only app is 1 to 3 requests per second, so memory is the metric to watch.
2. On a Mac resized to a CPX22, the stack peaked at 2,376 MiB of 4,096 with no OOM kills.
3. My own tool reported CPX12 as fitting while the API was dead.
4. A lighter identity stack would save about 0.5 GB, or €96 to €192 a year, and wouldn’t change the plan I buy.

Next: Part 2, why I rewrote a working .NET app in Go with Zitadel when my own research told me not to.