PoppyDB Goes Live β Measurements from a Real Bus
We have never been this far! PoppyDB went live. On our real message bus. Three nodes. MongoDB replaced in the same spot. The nastiest bugs we had already found and fixed during the test phase. Now the question was whether PoppyDB delivers in production what the tests promised.
The road here: 16 releases in two and a half months
Actually, the go-live day is not the whole story. The story is how we got there. Morphium 6.3.0 came out on August 9th. On October 21st, 6.3.15 was running on the bus. In between: sixteen releases.
Not because nothing worked. Sure, there were bugs too, especially under load or in failover cases. But because we fed every acceptance measurement straight back into a release, so we could test right away. Look at the changelogs: there is 6.3.2 and 6.3.3 on the same day. Then 6.3.4, 6.3.5 and 6.3.6 on a single day too. That was the phase where we worked through the findings. After that it got quieter, because the findings got rarer.
Think what you like about the pace. It worked well for us here. Every release was small. It contained only what came out of a concrete observation. No batch release with twenty things, half of which resurface later. You only get away with that with good test coverage. Luckily we put a lot of value into that in Morphium (and PoppyDB too). The suite with its 2,200+ tests had to stay green!
All these tests, the load profile, the measurements showed us: PoppyDB can save a lot in production and reduce load. Naturally, that only works in this special use case. An in-memory database is, by nature, limited by the available RAM. But our use case (a message bus) produces only little data, which just has to be passed on fast.
The swap
On the morning of October 21st, we took out the MongoDB replica set. PoppyDB came into the same spot. Same port. Same hosts. Same node count. The clients noticed nothing.
Now it was about finding out whether PoppyDB really works well in production. So we measured. Throughput, latency, errors, logs, resources. The baseline is the last full production day on MongoDB before it. Same times of day, so the comparison stays fair.
Throughput: more is always good
The first finding confirmed the hope from the lab. PoppyDB does not just keep up in production. It is more nimble and needs fewer resources.
In the valid window, where both days are cleanly comparable, that was roughly 13.6 percent more messages per hour than on the reference day. Hour by hour, PoppyDB sat between five and 27 percent ahead, with real data from a real production system.
So this does not stay abstract, a few absolute numbers. More than 500 topics run on the bus. On a normal working day that adds up to around 3 million messages. In the measured six-hour window it was just over 1.08 million on MongoDB and 1.23 million on PoppyDB. With that much traffic, 13.6 percent makes a real difference.
For context: the acceptance micro-benchmark came in at roughly 3.5Γ on the bus round-trip. That not mapping one-to-one onto production was expected. Here you measure real end-to-end behavior. But throughput goes up while latency goes down. Exactly what we had hoped for from a DB with less overhead.
Latency: less is more
Here are some concrete numbers. End-to-end latency across the bus, compared pairwise in the same six-hour window:
| Metric | MongoDB | PoppyDB | Ξ |
|---|---|---|---|
| Median (p50) | 60 ms | 39 ms | β35 % |
| p95 | 145 ms | 97 ms | β33 % |
| Mean | 78 ms | 47 ms | β40 % |
| Max | 1,818 ms | 300 ms | β83 % |
Three numbers stand out. The peaks dropped by 83 percent. That is the most valuable thing on a message bus. Those outliers are exactly what produces timeouts and follow-on failures. The p95 sits below 100 milliseconds. Before, it was 145. And the median is at 39 milliseconds.
Over the full 24-hour window it held: p50 at 40 ms, p95 at 96 ms, max 300 ms. It stays stable all day like that. 👍
Stability: fewer outliers
You do not see the biggest difference in the averages. It was exactly the problems with too much load in the old MongoDB system that PoppyDB was supposed to address. And the numbers show it clearly.
In the same window, the MongoDB day had 50 bus timeouts. PoppyDB in the same window: zero. The slow-query log lines dropped from nearly 30,000 to 17. That is about 1,760Γ fewer. And total server log volume shrank by a factor of 17.
Whether logs count as load, you can argue about at length. I would count them. A log that growls 30,000 times is not necessarily a "loud log". But it does put load on the system.
Over the following 25 hours: no outage, no restart, continuous heartbeat from all nodes. PoppyDB's own slow queries: 48 in 25 hours. mongod had roughly 30,000 in six. And nothing was ever rejected, not even at the 90-percent heap watermark.
Resources: relaxed
The nodes sat in the mid-single to low-double-digit percentage range on CPU on average over the day. With occasional spikes. Heap stayed constant over 24 hours at a mean around 5 GB. No leak.
What we saw there could maybe be improved. The G1 GC oscillates hourly up to near the 12 GB limit. The heap warning watermark rings regularly. Nothing was ever rejected. But that is exactly what we have noted for the next step: look deliberately at the buffer and heap budget. Instead of letting it collide by chance.
The caveats
As in the other posts, here are the limits of the numbers. The MongoDB reference day is a Tuesday, the PoppyDB day a Wednesday. Not identical weekdays, but close in operating profile. The morning numbers of the go-live day were not fully booked. So the throughput comparisons only rely on the hours from 11:00 onward. Slow query is not exactly the same format at mongod and PoppyDB. The curve is the same. And there was no direct benchmark on production before the swap. The micro numbers come from acceptance.
None of these caveats flips the direction. But do not take the percentages to the third decimal.
What remains
PoppyDB has arrived in production (yeah!). Throughput at least as expected, rather higher. Latencies a third to 40 percent lower. Peaks 83 percent lower. The "skid marks", meaning timeouts, slow queries and log growling, are practically gone. Stable over 24 hours. No leak. No restart.
And the road here was part of it. The 16 releases in two and a half months. Every number I can present today comes from an observation we once fed back.
PoppyDB and Morphium 6.3.15 are on Maven Central. The changelogs for the whole 6.3.x series are on GitHub.