Two Ways Out of SCADA, and the Vendor Picks
Every SCADA source falls into one of two access archetypes, decided by who administers the endpoint. The ingestion pattern follows from that.
By John Wassilak
A team settles on an ingestion architecture, builds it well, and then finds that half the estate cannot use it. The cause is structural rather than unlucky.
The cause is treating ingestion as a technology choice. On a heterogeneous estate each system falls into one of two access archetypes, the archetype is decided by who administers the endpoint rather than by what you would prefer, and the correct pattern follows from there.
graph LR
Q{Who administers<br/>the endpoint?} -->|A counterparty| G[Vendor-gated relay]
Q -->|You, or a partner| B[Direct broker]
G --> GP[Poll the interface they expose]
GP --> GN[Rate limits, pagination,<br/>access is a negotiation]
B --> BP[Subscribe at the edge]
BP --> BN[Session state, sequence gaps,<br/>durable landing is on you]
GN --> L[(Bronze)]
BN --> L
L --> M[One model downstream]
Archetype one: the vendor-gated relay
Some assets present their data through a system somebody else operates: a hosted SCADA platform, a service provider’s historian, a vendor portal with an API. The defining property is that the data is already concentrated somewhere convenient and you do not control the endpoint.
Consequences follow. Access is a commercial negotiation rather than a configuration change, so the timeline belongs to the counterparty. The interface is whatever they chose to expose, usually a REST API with pagination and a rate limit. You inherit their idea of a timestamp, their quality semantics, and their behavior under load.
Polling is correct here for a structural reason: you cannot push from a system you do not administer. Configuring that push would put you back in the access negotiation, for an efficiency gain on a link that is rarely the bottleneck.
The engineering risk is concentrated in one place. You are a guest on somebody’s production system, so everything in how not to take down your SCADA source applies with extra force, because the consequences land on a counterparty who was lukewarm to begin with.
Archetype two: the direct broker
Other assets you control. The gateway is on your network, or on a network your automation partner administers cooperatively, and it runs software that can publish.
Here the properties invert. Access is a configuration change on a machine you can reach. The edge decides what to send and when, so you can get report-by-exception semantics rather than polling on a fixed cycle. Discovery comes over the wire, because the publishing layer announces what metrics exist rather than requiring you to maintain a tag list by hand. Session state is explicit: you know when a publisher connected, when it went away, and whether you missed a sequence.
The engineering work moves to a different place. Instead of managing rate limits and pagination, you are managing subscription state, session continuity, and the ambiguity of silence, which the broker post covers in detail. You are also responsible for durable landing, because nothing behind the broker is keeping the data for you.
This archetype is better in most respects. It is also not available on demand, because whether an asset qualifies is determined by ownership and cooperation rather than by your architecture diagram.
The estate is always both
A multi-asset operator assembled through acquisition will have both archetypes in the portfolio, and the split rarely follows a tidy line.
It does not map to platform, because the same product can be self-hosted on one asset and vendor-hosted on another. It does not map to age, because a legacy asset with an in-house automation team can be more accessible than a modern asset on a managed service. It does not map to size or basin.
It maps to who holds admin on the box. That is the whole classifier, and it comes out of the provider mapping rather than the technology inventory.
Which means the archetype for each asset is a discovery finding, not a design decision. You do not get to choose. You get to find out.
What follows for the pipeline
The practical consequence is that a serious estate runs two ingestion paths and one model.
Two paths, because the shapes genuinely differ. A polling path with scheduling, watermarks, pagination handling, and rate-limit backoff. A subscription path with session tracking, sequence continuity, and durable landing. These are different code with different failure modes and different monitoring, and pretending they are one thing produces an abstraction that serves neither.
One model, because everything downstream of landing is identical. Tag normalization, unit conversion, quality flag translation, asset contextualization, and the canonical model do not care how a reading arrived. That is the whole point of the abstraction layer above the SCADA vendors, and the archetype distinction should not survive past the bronze layer.
Carry one provenance field, though: which path a reading arrived on. The two have different latency characteristics and different gap semantics, and it is the first thing anyone debugging a suspicious series six months from now will want to know.
The mistake in both directions
The failure to guard against is picking one archetype and forcing the estate into it.
Forcing everything into the broker pattern means building a bridge per vendor-gated system, polling the vendor and republishing so the downstream looks uniform. Sometimes that is right. Be clear about what it buys, though: the polling still exists, the rate limits still exist, the negotiation still happened, and you have added a hop for uniformity the silver layer would have given you cheaper.
Forcing everything into polling is the commoner error, usually because the first asset delivered happened to be vendor-gated and the pattern got generalized from a sample of one. Running a scheduler against an edge that would happily push is more infrastructure doing a worse job, and it throws away the discovery and session semantics that make the push path reliable.
Both are avoidable by classifying the estate before choosing the architecture.
The archetype can change, and usually improves
One more reason to keep the two paths genuinely separate rather than abstracting over them: assignments move.
A vendor-gated asset becomes broker-capable when the service contract is renegotiated, when the operator brings administration in house, or when the provider upgrades to a platform with publishing support. This happens more often than teams expect, because the commercial cycle runs on its own schedule and a contract renewal is exactly the moment to ask for data access terms.
If the ingestion paths are cleanly separated and everything converges at bronze, migrating an asset from one to the other is a swap of the ingestion component and nothing else. The tag mapping, the normalization, the asset hierarchy, and every downstream model keep working, because none of them ever knew which path the data arrived on.
If instead the polling logic has leaked into the transformation layer, which is what happens when one path is treated as the default and the other as a special case, that migration becomes a refactor. The place this shows up first is timestamps: polled data often carries the poll time alongside the event time, and models written against a polled source sometimes quietly depend on the former. Push data has no poll time at all.
Keep the boundary at bronze and the archetype stays an ingestion detail. Let it leak and it becomes an architectural commitment to a commercial arrangement that was never permanent.
Classify first
The sequence is short and it belongs before any pipeline design.
For each asset, answer one question: who administers the endpoint the data has to come out of? Your team, a cooperative partner, or a counterparty. That answer assigns the archetype.
Then build the archetype-two assets first. They are faster, they depend on nobody else’s calendar, and they produce the working pipeline that funds the rest. The archetype-one assets proceed in parallel on a commercial track at whatever pace access allows.
The classification took an afternoon on the estate we assessed, and it changed the sequence of the entire program. That is an unusually good return for a spreadsheet with one column in it.