Testing Failover Without Killing Anything — a Wire Proxy as Fault Simulator

The Problem: The Worst Outage Is the Silent One

If you build a database driver with replica set support, you have to test failover. Sounds simple: take away the primary, watch the driver recover. In practice, though, there are three very different ways a node can be "gone":

1. Clean exit — the process shuts down, the OS closes the connection properly (FIN). The driver gets an error immediately.
2. Hard death — kill -9, the port is closed, new connections are refused (RST). The driver notices this quickly too.
3. Silence — the machine freezes, the network partitions, a VM gets paused. The TCP connection is perfectly intact from the driver's point of view. There's just… nothing coming back.

Case 3 is the killer. A driver cannot distinguish a silent connection from a slow server — without its own timeouts it waits forever. And that exact case is the hardest one to test with conventional means.

Our first failover test was, frankly, manual labor: build a local replica set, kill -9 here, kill -STOP there, stare at logs. The test carried the tag manual and consequently never ran in CI. It did find regressions — but always only after someone had already hit them for real.

The Idea: Faults Belong on the Wire, Not in the Process

The aha moment: all three failure modes are, from the driver's perspective, pure wire phenomena. The driver never sees a process die — it only sees what happens (or doesn't happen) on its socket. So you don't need to kill any processes at all. You just need something that sits between the driver and a perfectly healthy replica set and breaks the wire in controlled ways.

One TCP proxy per replica set node:

Three fault modes, matching exactly the three failure types above:

freeze is deliberately implemented to be nasty: new connections are still accepted (the OS completes the handshake from its backlog, just like with a stopped process) — and then simply nothing happens. No error, no close, no data. Exactly like the real thing.

The Trick: Feeding the Driver a Fake Topology

A proxy alone isn't enough. Modern MongoDB drivers do server discovery: they ask any node via hello, and it answers with the full replica set topology — hosts, primary, me. All real addresses. After the first hello, the driver would happily bypass the proxies and connect straight to the real nodes, and your whole fault simulation evaporates.

The solution: the proxy parses the server's responses (backend→client direction) and rewrites the real addresses in every hello response to the proxy addresses. The driver now lives in a small, consistent alternate reality in which the replica set consists of three proxies — and every connection, including the ones discovery itself opens, flows through the fault gate. A dedicated test pins exactly that: after full connection setup, the driver must be connected to proxy addresses only.

A nice detail on the side: only server responses get parsed. Everything the client sends toward the server is forwarded as raw, length-prefixed frames — the less the proxy "understands", the less it can accidentally distort.

For the cases where you want to control the real replica set (a genuine replSetStepDown still triggers the most authentic election there is), there's a separate control channel straight to the real nodes — cleanly separated from the data path the driver sees.

What Came Out of It: Real Bugs, With Numbers

The beauty of a harness like this: it doesn't find theoretical bugs, it finds observable ones. A selection of what the suite flushed out of our connection pool on its first serious run:

• The unreachable deadline. borrowConnection() had a time limit — but it was only checked when the pool was empty. After a freeze, the pool was full of dead connections, the limit was never evaluated, reads hung. A classic case of "the timeout exists, but the code path never reaches it."
• Reads that don't recover even though writes do. The read-preference logic (NEAREST / PRIMARY_PREFERRED / SECONDARY_PREFERRED) never fell back to the healthy new primary after a failover. Observed live: a single countAll() spent 26 seconds hammering the dead ex-primary — while the freshly elected primary sat idle right next to it. Writes: long since fine. Reads: frozen.
• The "fastest host" that no longer exists. The NEAREST cache kept pointing at the frozen ex-primary for ~11 seconds after the failover — and every single read paid a full server-selection timeout before being allowed to go elsewhere.
• Waiting instead of switching on stepdown. When the old primary itself announces who the new one is during stepdown, the driver still stubbornly waited for the next discovery cycle instead of using that information.

Every one of these fixes now has a deterministic unit test — but they were all found by the proxy harness, under realistic conditions, in CI.

Bonus: One Harness, Two Servers

Because the proxy only speaks wire protocol, it doesn't care what's behind it. The same test suite runs against a real MongoDB replica set and against PoppyDB, our MongoDB-compatible in-memory server with its own leader election. The driver is checked for identical failover behavior against two completely independent server implementations — and as a side effect, the suite hardens PoppyDB's own election too.

Takeaways

• Failover bugs are almost always wire bugs — so simulate them on the wire.
• A TCP proxy with three fault modes (close, reset, freeze) covers the three real-world failure types; freeze is the one that hurts.
• Without hello rewriting, the prettiest proxy is useless — discovery will leak the real addresses otherwise.
• No kill -9, no sudo, no hand-built infrastructure: the whole thing runs as a normal test against any reachable replica set — and therefore finally in CI, instead of in the head of the one colleague who knows how to drive the manual test.

The code (WireProxy, AddressRewriter, the full failover suite) lives in the Morphium repository under morphium-core/src/test/java/de/caluga/test/morphium/testutil/proxy/ and ships with Morphium 6.3.0.