Morphium 6.3.1 - and the Transparent Message Bus

Two weeks after 6.3.0 we just released Morphium 6.3.1. On paper it's a bugfix release - in reality it contains the first two production weeks of 6.3.0, a new messaging feature and a couple of fixes that, honestly, nobody would have found without watching the message bus do its work live. And that's what I want to write about this time: why a message bus whose messages are simply documents in a database is a very different kind of system than Kafka and friends - for testing and in production. But one thing at a time...

What's new in 6.3.1?

Messaging: the change stream now filters server-side

The most important fix first. Until now, every consumer's change-stream cursor got to see every insert into the messaging collection - including messages addressed to completely different recipients, full payloads of large answers from other producers included. Under high traffic (for us e.g. during document-export bursts) the cursor fell behind, and messages only arrived via the fallback poll - with the corresponding latency.

The main change stream is now built with a server-side $match that only lets through what the instance can actually process: messages addressed to it, broadcasts for topics with a registered listener, answers. If the listeners change at runtime, the stream is rebuilt with the new filter. Small detail on the side: messages from V5 senders only have a name field instead of topic - the filter honors both, otherwise legacy senders would simply have been filtered out silently.

Same category, #286: the lock change stream's callback ran a countAll query per deleted lock - on the change-stream thread, of all places. During a burst of lock releases the stream basically blocked itself. The query is gone for good, a counter coalesces any number of lock events into exactly one poll.

Plus two robustness fixes from the code review: a failed stream rebuild is now retried on the next tick (before, the instance considered itself up to date and simply kept running without a main change stream), and the listener registry is no longer modified in place while the poll thread is reading it.

New: implementation mismatches are detected

Morphium meanwhile ships three messaging implementations: StandardMessaging, MultiCollectionMessaging and, since 6.3.0, DualChannelMessaging (beta). Their collection layouts are not compatible - and the worst part about that used to be: a mixed deployment failed silently. Broadcasts kept flowing, but answers ended up in a collection the other side never reads. "Most things work, only answers never arrive" is about the nastiest failure mode you can imagine.

As of 6.3.1, every messaging instance announces its implementation on startup in a layout-independent <queue>_participants collection (with a heartbeat) and checks what the other participants are running. The channel is deliberately not the messaging itself - between two implementations without a shared collection, the warning would never arrive 😉. On a mismatch you get a warning by default; if you prefer it strict:

With that, a misconfigured instance refuses to start instead of silently losing answers.

In-memory driver: the correctness offensive continues

6.3.0's motto was "green tests against the in-memory driver have to mean something". That continues in 6.3.1 - with three fixes that pack a punch:

• An index on an array field silently returned an empty result set for equality queries (#289). Multikey indexes are not implemented in the index store, but the query planner used such indexes anyway. What that means in a real system: a job scheduler selecting work with processed_by == "X" finds nothing - and releases nothing. Such indexes are now detected as multikey (even if the array sits in the middle of the path, i.e. {"a.b": 1} over {a: [{b: ...}]}) and taken out of planning. Correctness over speed.
• Change-stream events arrived out of order under load. The dispatcher handed each event to a thread pool as its own task - and a pool simply doesn't guarantee ordering. mongod guarantees event order per cursor, and now the in-memory driver does too (one dispatcher thread, unbounded queue, writers still never block). By the way, this only surfaced because a new regression test went sporadically red on the concurrently loaded test runner - on idle hardware the race had been invisible since March. You don't find bugs like that in a debugger.
• The update and replace operation types now match mongod exactly (#288). A replaceOne used to produce no event at all (i.e. the change was invisible to every watcher - including PoppyDB replication!), and a store() of an existing document reported replace where mongod reports update plus updateDescription for the underlying $set update. If you test your change-stream consumers against the in-memory driver and run them against MongoDB, you now get the same events in both worlds.

Also: collection and index creation are now atomic (two concurrent createUser calls could both win before), and PoppyDB replication applies events right when they arrive instead of waiting for a 5ms flush tick - noticeably less lag on the secondaries.

PoppyDB as a test system

Why all this effort for an in-memory driver in the first place? Because it's the foundation of PoppyDB - and PoppyDB is our test system.

The argument is simple: PoppyDB speaks the MongoDB wire protocol, starts in a few hundred milliseconds, needs neither Docker nor an installation, and behaves - thanks to exactly this correctness offensive - like mongod, down to event ordering and error codes. Our test matrix runs in five phases: in-memory, MongoDB replica set, PoppyDB replica set, MongoDB single, PoppyDB single - the same ~215 test classes against all backends. Any divergence between in-memory behavior and mongod is a bug. And #288/#289 show this is meant seriously: both were found because tests were green on one backend and red on the other - and both times the fix was not "adjust the test" but "fix the driver".

For your own CI this means: one poppydb artifact in test scope, one port, done. No container startup, no flaky networking. And if you want, you can test failover scenarios against a real 3-node replica set made of three JVM processes.

...and as a message broker in production

Now it gets more interesting: for messaging, PoppyDB is not just a test system.

Messages are transient by nature - Morphium messages have a TTL and get deleted after processing. Writing data that expires in seconds anyway to a write-ahead log is durability you pay for and never collect. That's exactly where a replicated in-memory store fits: a 3-node PoppyDB replica set covers node failures via replication and Raft failover, the memory watermark turns overload into retryable backpressure instead of an OOM, and since 6.3.0 there's SCRAM auth, TLS and configuration files on top.

The effect is measurable. Identical Morphium messaging code on both sides, both systems as a 3-node replica set on the same machine:

The gap is structural, not tuning: MongoDB change streams only emit majority-committed events - so every messaging hop pays replication plus journal commit. That's a latency floor of ~35ms, and it buys durability. PoppyDB emits straight from memory, because there simply is nothing to persist. You get the latency because you accepted the loss model: no WAL means a cluster-wide outage loses the in-flight messages. For events, cache invalidation and job triggers with sender-side retry that's the right deal. For guaranteed delivery it's the wrong tool - and by the way, the docs say exactly that, i.e. nothing is being sugar-coated here.

On top of that comes something a generic store can't offer: PoppyDB knows Morphium messaging. Instances register their messaging collection with the server (registerMessagingCollection), and the server then pushes e.g. lock_released events over the main change stream - the client needs no second cursor on the lock collection, and exclusive messages get redistributed event-driven instead of poll-driven. The broker optimizes for the protocol running on it.

The transparent bus

Which brings us to the actual point. The biggest difference between this messaging and a classic broker is not throughput or latency - it's that the bus is transparent. Messages are documents in a collection. You can query them, aggregate over them and watch them via change stream, without the messaging noticing anything.

Reading along without consuming

I already described the pattern in the post about our enterprise message bus, but it belongs front and center here. We run a MessageBusPeeker: a service that opens a change stream on the messaging collection and reads along with everything - which request came when, whether and when its answer arrived, round-trip times per message type, and which requests had no answer after two minutes. Out of that come hourly and daily time series per topic: volume, response times, unanswered rates.

The crucial part: the peeker consumes nothing. It acknowledges no messages, moves no offsets, takes part in no rebalancing and does not affect delivery in any way - senders and receivers don't even know it exists. It's simply another MongoDB client with read access to a collection. With Kafka you'd need a dedicated consumer group with its own offset management for this - here it's one change stream and a hundred lines of code. And the new server-side topic filter changes nothing about that, by the way: it filters the consumers' cursors, an observer attaches to the same collection with its own pipeline.

Morpheus: the cockpit for the running bus

What the peeker does as a background service, Morpheus does interactively: a terminal UI (plus a scriptable CLI) that attaches to the running bus. Production or test, MongoDB or PoppyDB - thanks to the wire protocol it's the same motion.

• messages - a top-style live monitor: sender, topic, processing state, exclusivity, answer with round-trip time, timeout highlighting.
• topics and nodes - aggregates per topic and per sender/answerer pair: volume, average RTT, timeouts. Load imbalance between nodes shows up here first.
• status - a per-node health monitor (heap, cache hit ratio, connections, errors, threads), with a drill-down into the full status dump. Built on the morphium_status topic every messaging instance answers anyway.
• graph - the message flow as an animated graph: nodes on a ring, messages as topic-colored shots, timeouts in red. Admittedly: nobody needs this, but it looks fantastic 😉
• latency - live latency graph and load test in one: interval, fixed-rate and ramp modes, a percentile table per responder, plus pong as the echo responder for the other side. The PoppyDB numbers above come from exactly this harness (latency --headless writes JSON/CSV/Graphite).

The point is not any single feature. The point is that all of this exists against the running system - no agent, no instrumentation, no deployment change. morpheus messages -c prod, and you see what the bus is doing right now. Several of the fixes in 6.3.0 and 6.3.1 - the falling-behind cursor behind #283, the lock stalls behind #286 - were found exactly this way: not in a debugger, but by watching.

This observability is not a bonus feature, it's a direct consequence of the architecture: because the queue is a database and not a log, every database tool automatically is a bus tool too.

Installation

Morphium 6.3.1 is available via Maven Central as always:

And for tests (or the broker):

No API changes, no new dependencies, no migration effort coming from 6.3.0. Issues #280, #283, #286, #288 and #289 are closed with this release, details as always in the CHANGELOG on GitHub.

If you run 6.3.0 with messaging on PoppyDB, upgrade promptly - the change-stream fixes (#288, event ordering) affect replication there too. Everyone else gets a release that above all does one thing: make sure that what the bus reports is actually true.