Your MQTT Broker Is Not a Historian
A broker holds days, not years. Sparkplug birth certificates and report-by-exception change what silence means for your pipeline.
By John Wassilak
“We already have MQTT, so we already have the data.”
That sentence is half right in a way that costs people years of history. The broker is moving the data. Keeping it is somebody else’s job, and frequently nobody’s.
What a broker is for
An MQTT broker is a router with a buffer. Publishers send, subscribers receive, and the broker holds messages long enough to deliver them reliably to clients that were briefly disconnected.
The retention features exist to serve that job, and each comes with a limit worth knowing.
Persistent sessions let a subscriber that dropped off reconnect and collect what it missed, but only for QoS 1 and 2 subscriptions with a non-clean session. QoS 0 messages are never queued, and QoS 0 is what a lot of edge software publishes by default.
Retained messages hold the last value on a topic so a new subscriber gets current state immediately. Note that this is unavailable in the case most of this post is about: Sparkplug requires the retain flag be false on its data messages, which is precisely why it needs birth certificates instead.
Queue depth and message expiry bound how much the broker holds. Message expiry is an MQTT 5 property and does not exist in 3.1.1, which plenty of industrial edge software still speaks.
Every one of those is a delivery mechanism rather than an archive. And when the queue fills, check which end your broker discards from before assuming: the common defaults drop the newly arriving message rather than evicting the oldest, which means an ingestion outage costs you the most recent data, not the stalest. That is the opposite of the mental model most people carry, and it is a per-broker configuration choice almost nobody has looked at.
Either way the buffer is sized for outages measured in minutes to days, because that is what it is for. A generously configured replay window runs to about a week. A week is not history.
The switch that eats your history
Here is the concrete version of the problem, and it is not hypothetical.
An operator changes SCADA providers, or an asset moves between service firms, or a hosted platform contract ends. The new arrangement stands up cleanly, data flows, and everything works. Some months later somebody wants to compare this quarter against the same quarter two years ago, and discovers the pre-transition history is gone.
Not corrupted. Gone. It lived in the previous provider’s platform, the contract ended, and nobody’s job was to extract it first. The broker held a week of it, and the week expired long before anyone asked.
No amount of connector work recovers that. This is the strongest practical argument for landing raw operational data in storage you own from day one, and it has nothing to do with analytics. It is a continuity argument. The analytics are a bonus.
That is also the argument for an open format on your own object storage rather than inside whichever platform is current. Data you can read with any engine, sitting in an account you control, survives a vendor change by default rather than by project.
What Sparkplug adds, and what it does not
Plain MQTT is a transport with topic strings and payloads. It says nothing about what the payload means, what metrics exist, or whether a publisher is still alive. Sparkplug B is the specification layered on top that fills those gaps, and it is worth understanding because three of its mechanics change how you ingest.
Birth certificates. When an edge node or device comes online it publishes a birth message enumerating its metrics, with datatypes and metadata. This is tag discovery arriving over the wire rather than out of a spreadsheet, and it is genuinely useful: your ingestion can learn what exists instead of being told.
Death certificates. A node registers a death message as its MQTT will, so if the connection drops uncleanly the broker publishes it on the node’s behalf. You find out a publisher went away rather than inferring it from silence.
Metric aliases. The birth message assigns each metric an integer alias, and subsequent data messages may carry only the alias, with the metric name omitted entirely. This is the mechanic that settles the argument below: a subscriber that missed the birth cannot decode which metric a value belongs to. Not degraded context. It cannot resolve the name at all.
Report by exception. After birth, publishers send changed values rather than every value on a cycle. The specification words this as SHOULD rather than MUST, and a fair number of implementations publish cyclically anyway, so confirm what your edge software actually does. Where it is honoured, this is why Sparkplug is efficient over constrained links, and it is the mechanic that misleads data teams most.
The mechanics above are from Sparkplug B, currently at 3.0.0 under the Eclipse Foundation, topic namespace spBv1.0. Check the version your edge software implements before building against the detail.
Report by exception makes silence ambiguous
Under report by exception, no message does not mean no change. It means no change was reported, and those are different for reasons that matter to a pipeline.
A tag that has published nothing for six hours might be perfectly steady. It might also be on a device that dropped off, or one whose publishing stalled, or one that was removed from the configuration. The absence looks identical in all four cases.
This is what the birth certificates and sequence numbering are for. Sparkplug messages carry a seq so a subscriber can detect a gap, and a birth resets it for a new session. Two details matter in practice: seq is an eight-bit counter that wraps at 255, so gap detection works modulo 256 and a subscriber that missed more than 256 messages cannot tell how many it lost; and the death message carries no seq at all, only the session counter. A subscriber tracking sequence continuity knows the difference between quiet and broken. One that just writes payloads to a table does not.
The practical consequence for ingestion is that you cannot treat the message stream alone as sufficient. You need the session state: which nodes are currently born, what sequence you last saw, when the last birth was. Without it, the pipeline cannot distinguish a healthy steady signal from a dead one, and a freshness test written against last-message-received will alert constantly on exactly the tags that are behaving best.
There is a related modeling consequence. Report-by-exception data is sparse and irregular by construction, so a naive resample to fixed intervals will do the wrong thing at the edges unless you carry values forward deliberately and record that you did. That decision belongs in the silver layer alongside the other gap-handling rules, made once rather than per dashboard.
Four things to check on your own deployment
If a broker is already running somewhere in your estate, these are answerable this week.
What is the actual retention. The configured value is the ceiling, not the answer. Queue limits, message expiry, and disk pressure interact, so a broker under load discards earlier than its nominal configuration suggests. Find out what happens when the disk fills, and which end of the queue goes first.
What happens when the subscriber is down. If your ingestion process stops for six hours, does the broker hold that data for you or not? The answer depends on whether the session is persistent and what the queue depth is, and plenty of deployments discover during the first real outage that they were using a clean session and lost the window.
Whether anything downstream is durable. Trace the path from broker to storage and find the first thing that survives a restart. If the answer is “the warehouse, on the next batch,” then everything between the edge and that batch is at risk for the length of the batch interval.
Who else is subscribed. Brokers accumulate consumers. A subscriber nobody remembers deploying, pulling every topic and writing somewhere unmonitored, is both a load problem and a data-governance problem, and it is invisible unless you go look at the connection list.
Where the broker fits
None of this is an argument against MQTT. Where the edge supports it, publish-based ingestion is a better pattern than polling, and Sparkplug’s discovery and session semantics solve real problems that plain protocols leave to you. The unified namespace conversation is worth having on top of it.
The point is narrower. The broker is the delivery mechanism, and something behind it has to be the record.
That something should be storage you own, written to continuously, with the raw payloads preserved before any interpretation. Once that exists, the broker’s retention window stops mattering, because the broker is only ever bridging the gap between the edge and durable storage. A week of buffer is plenty for that. It is nothing at all for a history.
Ask what your replay window is. Then ask what happens to everything older than it when the contract changes.