Staleness is a value, not an absence

#The one-line fix

A vibration sensor on a pump goes quiet at 02:10 and returns at 03:40. You have a reading before and a reading after. Interpolating between them is trivial and the chart stops having a hole in it.

Somebody looks at that chart to decide whether the pump needs attention. A straight line across ninety minutes is a claim about what the pump was doing, and you do not have that information. The chart looking correct is the mechanism by which the wrong decision gets made.

#Making the gap a state

Every channel carries an expected interval as part of its definition. A one-second channel is expected every second, a daily meter read is expected daily. Once the gap since the last observation exceeds a multiple of that interval, the value's state becomes stale.

Stale is not null and it is not the last known value on its own. It is the last known value, the time it was observed, and an explicit flag, and every read path returns all three. There is no accessor that hands back the number alone, which made a few early call sites uglier than they wanted to be.

1type Reading struct {
2    Value      float64
3    ObservedAt time.Time
4    State      ValueState // Live | Stale | Unknown
5}
6
7// There is no Reading.Float() that drops State on the floor. That was
8// deliberate, and it caused three arguments.
Copy

Industrial protocols already model this. An OPC UA value carries a status code for exactly this reason [5], and a pipeline that discards it is discarding information the field devices went to the trouble of providing.

#Out of order is the normal case

I designed the first ingestion path assuming observations arrive roughly in the order they were taken, with occasional stragglers. On a real site, four things happen instead.

• Edge gateways buffer during an outage and flush several hours at once, in order among themselves and very late relative to everything else.
• Two collectors on the same historian, configured for redundancy, deliver a meaningful fraction of observations exactly twice with identical timestamps and values.
• Clock skew on field devices produces observations stamped slightly in the future.
• A historian backfill arrives as three months of data on a Tuesday afternoon at a rate nothing like the live stream.

Late data is not an error path. On a real site it is a double-digit percentage of the total, and any design that treats it as exceptional will spend its life in the exception.

#Event time, and a bounded window

The framing that resolves this is the separation between event time and processing time, which the streaming literature settled some time ago [1] [4]. State is a fold over observations keyed on when they happened, not when they arrived, so replaying in any order converges on the same answer.

Sorting still needs a window, and the window has to close or nothing is ever emitted. Sizing it per channel from that channel's own measured lateness works considerably better than one global constant, which is the same conclusion the out-of-order stream processing work reached [2] [3]. A satellite-linked site and a wired unit differ by orders of magnitude, and any constant chosen for one makes the other either laggy or permanently in correction.

Observations outside the window are still accepted and stored. They land as corrections, which re-project their window and propagate forward. Nothing is discarded for being late.

#What I would do differently

I spent too long looking for one window that worked everywhere. Deriving it per channel from measured lateness was obvious in hindsight and took me about two months to reach.

I also wish the staleness flag had been there from the first version rather than added when somebody asked why a decommissioned sensor was still reporting 42.3. Retrofitting an extra state into every read path is exactly as tedious as it sounds.