Morphium 6.3.4 to 6.3.6: Fixing Bugs Nobody Ever Hit

Morphium 6.3.4, 6.3.5 and by now 6.3.6 are out on Maven Central. Three releases within two days, the fifth within a week. And this one is different from all the others: not a single fix in it goes back to a bug report. Nobody hit these bugs. No stack trace from production, no issue, nothing. All of them were found by pure code review - done by AIs.

And yes, I have feelings about that. Mixed ones.

AI review - why though?

I let AI agents loose on the change-stream and replication code. Deliberately not one model but several: OpenAI, Anthropic and opencode, in collaboration. Each one reviewing on its own, from different angles. Two of the findings even came with a note that two independent agents had found the same bug from opposite ends - one coming from the change-stream side, one from replication consumption. When two reviewers with different training land on the same line of code, you'd better take a look.

What they found is a bug class my test suite is simply blind to: bugs that only exist under load. Not the "this assertion is wrong" kind. More the "the timing window between these two threads silently eats your data" kind.

So what was actually broken?

A taste, all fixed in 6.3.4:

A resume check that checked too early. Whether a change stream can be resumed was validated at registration - but the actual replay ran later, on another thread, while eviction happily kept running on every write. A resume validated as "clean" could lose events in exactly that gap. The consumer then got the rest delivered, with an invisible hole in the middle.
Live events overtaking the history replay. A live event with sequence 105 could arrive before the replayed 101 to 104. On a replica secondary that can overwrite a new document with an old state. Silently, of course.
10,000 times 300KB is 3GB. The watch-cursor queues were bounded by count only. A single slow consumer with large documents could pin gigabytes on the primary. On exactly the node whose OOM takes the whole cluster down.
An initial sync declaring "success" over a dead watch. Backpressure parks the watch reader during the sync - and the parked reader is, annoyingly, the only thread that could notice the primary killed the cursor long ago. The sync gate happily opened anyway.
A stopped sync thread writing into its successor's data. stop() waits 5 seconds for the thread, but the socket read waits up to 60. The orphaned thread resurfaced later with a finished read in hand and just wrote data that meanwhile belonged to its successor.

The pattern is always the same: silent. No exception, no log line, no red test. Just data that is quietly wrong, waiting for load profile and failure mode to line up in production some day.

And in 6.3.5 we had to fix a production bug from 6.3.4 right away - again one that only occurs under load. When the resume token a client sends is "out of scope" for the server, the server answers with code 286 ChangeStreamHistoryLost. Stream ends, client restarts fresh - that would have been the expected behaviour.

Unfortunately, due to a simple programming bug, the token was reused and sent again. So again the 286, again a restart, again the same token... endless loop. That was hammering the MongoDB and PoppyDB servers to a standstill - a self-made DDoS.

And 6.3.6? That one was the creepiest of the lot. During a rolling restart of the servers, exactly one client out of about thirty just stopped: no more messages, no exception, not a single log line - while its HTTP side stayed perfectly healthy, all health checks green. A thread dump later it was clear: every heartbeat thread of the driver was gone. The cause, after some digging: the topology bookkeeping registered hosts under their normalized name, but the cleanup compared against the un-normalized names from the server hello. If a server advertises a member in different case - SERV-MSG1 vs serv-msg1 - the driver considered the very host it had just added "no longer part of the replica set" and threw it out. A few such hellos in a failover window, and the whole topology including the seed list was eaten empty. After that: silence, forever. One uppercase letter, one dead client.

A proper release storm, too many for my taste. But they were necessary, and the upside is real: the 2000+ test cases could never model the load of a real, busy system before. Now they can - every one of these bugs got a regression test, and the qualification run for 6.3.6 was the first of the entire week that went through without a single retry.

The love-hate part

The hate first: none of these bugs ever happened. The code ran. The suite was green. Everybody happy. And then a bunch of language models hands me a stack of meticulously argued findings, with line numbers, race windows and worked failure scenarios - and suddenly I have a week of work I never ordered. Verification, regression tests, fixes across three modules, two full test-matrix runs, and a benchmark on top to prove the fixes cost nothing (they don't, throughput is unchanged). That's real work for bugs that, from the outside, didn't even exist. Thanks a lot.

Now the love: I know, of course, that this view is nonsense. These are the worst bugs there are, precisely because they haven't happened yet. Silent divergence doesn't file a bug report. It shows up months later as "the data on that node looks weird", and good luck hunting down the cause then. By the way, every finding was verified against the code before a single line was touched - these were no hallucinations. This was the kind of review a very patient, very pedantic colleague would do if he had a week per file. Which nobody has.

One finding earned itself a special place: a test that CI had considered "flaky" forever wasn't flaky at all. Its sporadic failure was a real bug - a hard integer cast in the messaging poll path that dies on 64-bit numbers. Of all paths, the one messaging uses to recover after a failover. The "flakiness" was the bug, politely knocking for months. We just kept hitting retry 😉

What now?

The review didn't just produce fixes, it produced homework. There are a few new issues on the 6.4.0 milestone now: a test harness that works with invariants instead of counting documents (because a suite that counts sees none of the above), a rework so there's no longer one parked thread per change stream, and the rest of the sync-shutdown hardening.

Heavy week. Five releases, a good dozen fixes with a proper root cause each, several green full runs of the test matrix, one benchmark. All for bugs nobody ever hit. Yet. And that "yet" is exactly why I'll do this again. Grumbling, but I'll do it.

Morphium 6.3.6 is on Maven Central, the full changelog is on GitHub.