--- title: "Choosing time series storage is really choosing a retention policy" date: "2023-10-01T05:00:00.000Z" categories: "Data" canonical: "https://xraph.com/writing/time-series-storage-is-a-retention-decision" --- # Abstract The engine matters less than what you decide to keep and at what resolution, because that decision is irreversible and gets made implicitly by default settings. ## The question that gets asked Which time series database should we use. It is the wrong first question, and it gets asked first because it is the one with vendor material behind it. The decision that determines whether the system is useful in three years is what you retain, at what resolution, for how long. That is irreversible in one direction: you can always aggregate later, and you can never recover resolution you discarded. ## What actually drives the volume Cardinality, not sample rate. A hundred thousand points at one sample per second is a predictable, compressible load. The same hundred thousand points, each tagged with several dimensions that vary, is a different problem, because most engines index by series and a series is the combination of all its tags. The failure mode is specific and I have watched it happen twice. Somebody adds a tag whose value varies per reading, such as a request identifier or a batch number. Series count explodes, the index no longer fits in memory, and query latency goes from milliseconds to seconds with no change in data volume. ## The tiers that work A pattern that has held up across several systems: - Full resolution for a short window, measured in days to weeks. This is what operational queries and incident investigation actually use. - Aggregated resolution for the medium term, measured in months to a couple of years. One minute or five minute buckets with min, max, mean and count, which answers most trend questions. - Full resolution retained indefinitely around events. When something happened, keep the raw window either side of it, because that is the data somebody will want later. That third tier is the one people leave out, and it is the one that matters. Downsampled data is adequate for trends and useless for understanding a specific failure. ## Aggregates you cannot recompute Min, max, sum and count compose. You can compute a daily maximum from hourly maxima. Percentiles do not compose: a daily 95th percentile cannot be derived from hourly 95th percentiles, and if you did not store a sketch you cannot recover it. Decide up front which non-composable statistics matter and store the structure that supports them. This is a well understood constraint in stream processing [2], and it is regularly rediscovered by teams who downsampled first and were asked for percentiles second. ## Separating hot from cold Recent data is queried constantly and old data is queried rarely and must remain available. Putting both on the same storage means paying fast storage prices for the majority of the volume. Separating compute from storage is the general form of this and is why cloud database designs put durable data on cheaper shared storage with a caching tier in front [3]. For time series the same split is easier, because the access pattern correlates almost perfectly with age. ## Where the boring answer wins Postgres with a time series extension handles more than people expect, and it brings relational data, transactions and a query language the team already knows [1]. Adding a specialised engine adds an operational unit and a second consistency story. Reach for the specialist when you have measured a specific limit, not because the workload has the word telemetry in it. # References 1. Martin Kleppmann, "Designing Data-Intensive Applications", O'Reilly Media, 2017 2. Tyler Akidau, Slava Chernyak, Reuven Lax, "Streaming Systems: The What, Where, When, and How of Large-Scale Data Processing", O'Reilly Media, 2018 3. Alexandre Verbitski et al., "Amazon Aurora: Design Considerations for High Throughput Cloud-Native Relational Databases", ACM SIGMOD International Conference on Management of Data, 2017 --- *Unpublished draft. Not peer reviewed and not submitted to a venue.*