Morphium 6.2.6: The Timeout That Multiplied Itself by 100

Morphium 6.2.6 is on Maven Central. By version number it's a patch release — by content it's the most important 6.2.x so far. Because the core question this release answers is: what actually happens when the primary of a replicaset dies? Until now, the honest answer was: nothing good.

The experiment: kill -9 the primary

The trigger was a production incident: primary gone, and the service never recovered. Not slowly — not at all. Writes hung, messaging stayed dead, only restarting the application helped.

The contract of a replicaset driver is clear enough: if the primary dies — crash, frozen VM, network partition — the replicaset elects a new one within a few seconds, the driver notices, reconnects, and the application keeps running. So that's exactly what we reproduced: a local 3-node replicaset, sustained write and messaging load, and then we did to the primary everything real life has in store — SIGTERM, kill -9, SIGSTOP (a frozen VM looks exactly like a network partition to a driver), restart while the primary is down.

The result before 6.2.6: after a hard kill, one single successful write in 45 seconds, messaging permanently dead. The failover the replicaset had completed in ~10 seconds simply never reached the driver. And there wasn't one cause — there was a chain of bugs, each one hiding the next.

Bug #1: readNextMessage × 100

The main character. When reading a reply from the server, readNextMessage tolerated up to 100 consecutive socket timeouts before giving up. Sounds like robustness, but it's arithmetic: the configured timeout is effectively multiplied by 100. maxWaitTime = 60s becomes more than an hour — per operation, per connection.

In practice that means: every operation in flight on a connection to the dead primary didn't hang for 60 seconds, it hung essentially forever. No error, no retry, no failover — just threads patiently waiting for a host that no longer exists. The timeout is now a hard total deadline.

Bug #2: The step-down detection that never fired

When a write lands on an ex-primary, MongoDB replies with a "not primary" error, and the driver is supposed to retry the write on the new primary. Morphium had that code — it compared the error message against the string "not primary". The actual message reads "Error: 10107 - not primary". The comparison could never match. Classic dead code that looks like error handling.

Step-downs are now detected via the mongo error codes (10107, 189, 91, 11600, 11602, 13435) and the write is retried on the freshly resolved primary. Network errors and missing replies are handled the same way — at-least-once, like retryWrites in the official driver.

Bug #3: One "No such host" and messaging was dead for good

Morphium's messaging runs on a change stream. The ChangeStreamMonitor treated the "No such host" exception as fatal and terminated permanently — and that exception occurs perfectly routinely in the short window during failover when a host has been evicted from the host list and not yet re-added. One unluckily timed error, and messaging was gone until the application restarted. Now: retry instead of surrender, and the error handling is extracted into a testable handleWatchError().

And the rest of the chain

There's more, in short form:

• Primary discovery via secondaries failed whenever the primary advertised in hello differed from the entry in the host map by casing or port spelling — the comparison ran without normalizeHostKey.
• Dead hosts were detected too slowly: heartbeat hellos and the connect handshake used maxWaitTime instead of a tightly bounded timeout. Eviction now also closes borrowed connections, so in-flight operations fail fast and get retried instead of running into bug #1.
• SingleMongoConnectDriver slept sleepBetweenErrorRetries * 10000 milliseconds when reconnecting after a null hello — a typo that turned seconds into ~16 minutes.

Each of these bugs alone would be annoying. Together they formed a system in which the failover path simply never ran to completion: whoever got past bug #2 hung in bug #1; whoever escaped that lost messaging to bug #3.

After

The same test suite after the fixes: after a hard kill of the primary, the application is back to full write throughput ~25 seconds later, with no lost messages — across all scenarios, including the freeze and the restart-while-primary-down case. Recovery time essentially scales with connectionTimeout plus the replicaset's own election time.

The scenarios live on as a manual test suite in the repo (FailoverReproTest), so this path never rots untested again. Which is the actual lesson here: failover code is code that almost never runs — and has to work precisely when everything else is on fire. Code that only executes during a disaster is effectively untested unless you regularly cause the disaster yourself.

One practical warning that surfaced during testing: entities with @WriteSafety(level = SafetyLevel.WAIT_FOR_ALL_SLAVES) will, on a degraded replicaset (one node missing), wait server-side on every single write until the configured timeout expires. If you have that in production, check whether MAJORITY is what you actually meant.

Also fixed: findOneAndUpdate deleted documents

The second fix in this release that deserves its own section — because it's about silent data loss: for entities with @Cache(readCache = true), findOneAndUpdate(Map) deleted the matched document instead of updating it on a cache hit (#214). The read-cache branch had been copy/pasted from findOneAndDelete() — including the delete.

The fix is conceptual: a find-and-update always has a write side effect, so it is never served from the read cache anymore. The command always goes to the database, and a successful modification invalidates the type's read cache.

Everything else in 6.2.6

SingleMongoConnectDriver

• dropCollection could deadlock against itself: the writer held the single connection while polling exists(), which wanted to borrow a connection of its own — minutes of stall whenever the dropped collection actually existed. The connection is now released before polling; on top of that, the connectionInUse flag is now an AtomicBoolean claimed via compareAndSet instead of a racy plain boolean (#215).

InMemoryDriver

• findAndModify now supports $setOnInsert and the upsert/new flags — matching real MongoDB, including a regression test for upserts with an $and-nested _id filter (#202, #203, #204).
• Dotted field paths: find no longer rewrites dotted query keys (paths with upper-case segments now match correctly), and distinct resolves dotted paths into the nested document.

Cleanups

• SequenceGenerator: ~40 duplicated lines of lock lifecycle extracted from getNextValue()/getNextBatch() into a shared withSequenceLock() — no behavioral change (#171).
• Legacy Vector/Hashtable in the cache, cache synchronizer and JMS producer replaced with ConcurrentHashMap/CopyOnWriteArrayList; listener iteration is now safe against ConcurrentModificationException. Remaining printStackTrace() calls now go through SLF4J (#173, #212).

Test infrastructure

• New manual test tag: the real failover tests (the ones that kill local mongod processes) are now cleanly separated from the CI-safe external tests and can never leak into CI again.

Upgrade

6.2.6 is a drop-in upgrade from any 6.2.x — no API changes. If you run a replicaset, or use @Cache(readCache = true) together with findOneAndUpdate, don't wait long. Full notes are in the GitHub release.

As always: feedback, issues and PRs are welcome on GitHub. Happy coding!