Series-Based Observability: Building Long-Term, Multi-Dimensional Observability Systems

1. Why a Single Snapshot Cannot Describe a System

Classical landscape painting includes a distinctive way of working: the same subject is painted repeatedly under different times of day, weather conditions, and viewpoints, and the differences themselves become the core of the work. The premise behind this approach is that any image captured at a single moment is a condition-dependent function rather than the subject itself. To understand the subject, one must understand its distribution across the space of conditions.

The same judgment applies to operations and data engineering. A single sample point on a latency curve can only answer, “What is the value right now?” A single threshold check can only answer, “Is the current value outside a predefined boundary?” Neither can answer the more important questions: Is degradation sudden or gradual? Is the impact broad or localized? Is the current fluctuation still within the normal range observed historically?

Treating long-window observation as a first-class concern rather than an afterthought usually requires three things at the same time:

• Enough multidimensional collection.
• Enough long-term retention with controllable precision.
• An alerting model based on distributions rather than point values.

The following sections examine these requirements layer by layer.

2. Collection Layer: Breaking “Lighting Conditions” into Observability Dimensions

A single dimension inevitably leads to misjudgment. Four commonly used core signals form the minimum viable dimension set:

• Latency: the distribution of request durations, not the average.
• Traffic: request volume or throughput over time, used to explain fluctuations in other signals.
• Errors: the proportion and classification of failed requests, distinguishing protocol errors, dependency timeouts, and business rejections.
• Saturation: the degree to which resource usage approaches its limit, which is the main basis for predictive judgment.

The value of having all four dimensions is cross-interpretation.

• If latency increases together with saturation, the likely cause is a capacity bottleneck.
• If latency increases while traffic decreases, the likely cause is upstream degradation or a client-side retry storm.
• If the error rate increases while latency remains unchanged, the issue is more likely to be a fast-failure path, such as authentication failure, quota exhaustion, or connection pool exhaustion, rather than a slow query.

If any dimension is missing, this kind of attribution degrades into guessing.

The collection layer also requires an immediate engineering constraint: control metric label cardinality. If dimension combinations are allowed to be concatenated without restriction, such as user IDs, request IDs, or full URL parameters, the number of time series can expand multiplicatively and overwhelm the storage and query layers. In practice, high-cardinality fields should be placed in logs or trace context, while the metrics side should retain only a limited set of enumerable labels.

3. Recording Method: Store Distributions with Histograms, Not Only Quantiles

Latency is a distribution-type signal. The mean has almost no diagnostic value because a small number of long-tail requests can distort it completely, and those requests are often the main contributors to poor user experience.

However, computing quantiles point by point on the client side is the wrong approach. The reason is explained in the section on quantile aggregation. The correct approach is to record bucketed histograms on the server side, aggregate them on the server, and calculate quantiles later in the storage layer when needed.

# Example: record latency with logarithmic bucket boundaries to balance tail resolution and series count.
LATENCY_BUCKETS_MS = [1, 5, 10, 25, 50, 100, 250, 500, 1000, 2500, 10000]
class LatencyHistogram:
    def __init__(self, buckets):
        self.bounds = buckets
        self.counts = [0] * (len(buckets) + 1)  # The last bucket is +Inf.
        self.sum = 0.0
        self.total = 0
    def observe(self, value_ms: float, ts_ms: int, tags: dict):
        for i, b in enumerate(self.bounds):
            if value_ms <= b:
                self.counts[i] += 1
                break
        else:
            self.counts[-1] += 1
        self.sum += value_ms
        self.total += 1

The choice of bucket boundaries directly determines the upper limit of quantile resolution. For example, p99 can only fall on a bucket boundary and cannot be interpolated within a bucket. For latency-sensitive services, bucket widths should be densified around expected quantile values. It is also useful to run a regression check on bucket distributions after each release. A rightward shift in the overall distribution often appears before changes in the error rate.

4. Retention Strategy: Layered Downsampling Instead of One-Size-Fits-All Retention

Keeping all data at raw precision for the full retention period is usually disproportionate to the business benefit. Keeping only aggregated values, on the other hand, permanently loses the ability to troubleshoot individual incidents. A common approach is three-layer retention:

• Raw-precision layer: complete resolution as produced, retained for hours to days, used for recent incident investigation and detailed root-cause analysis.
• Downsampled layer: minute-level and hour-level aggregations retained for weeks to months, used for trends, capacity planning, and periodic comparison.
• Aggregated-metrics layer: high-compression statistics such as counts, sums, quantile buckets, and key event milestones, used for macro-level evaluation over very long periods.

def rollup(points, step_sec: int, keep_sum=True):
    """Aggregate a (ts, value) series into fixed-size steps.
    Preserve count and sum instead of only the mean. For quantile buckets,
    keep bucket counts rather than precalculated quantiles.
    """
    out = {}
    for ts, v in points:
        key = (ts // (step_sec * 1000)) * step_sec * 1000
        slot = out.setdefault(key, {"count": 0, "sum": 0.0})
        slot["count"] += 1
        slot["sum"] += v
    return {k: (v["sum"] / v["count"], v["count"]) for k, v in sorted(out.items())}

One commonly overlooked pitfall during downsampling is this: counter-type metrics must be differenced, not averaged. Request totals are monotonically increasing within a time window. Taking an average inside the window erases the cumulative nature of the metric. Similarly, when downsampling, preserve interval maximum or minimum values when needed, such as peak saturation within a window, because averages systematically underestimate spikes.

5. Key Constraint: Quantiles Cannot Be Added or Averaged

This is the most frequent source of errors in long-term observability. Quantiles are nonlinear statistics. Different shards contain different sample sets, so their quantile values are not additive.

# Each shard contains 100 request samples.
node_a = [10.0] * 99 + [1000.0]   # Within-shard p99 is approximately 1000 ms.
node_b = [10.0] * 99 + [200.0]    # Within-shard p99 is approximately 200 ms.
# Incorrect approach 1: average the quantile values -> about 600 ms.
avg_of_p99 = (1000.0 + 200.0) / 2
# Incorrect approach 2: sum the quantile values -> 1200 ms, also meaningless.
sum_of_p99 = 1000.0 + 200.0
# Correct approach: merge all 200 raw samples and recompute the quantile.
merged = sorted(node_a + node_b)          # 198 values of 10 ms, followed by 200 and 1000.
p99_of_union = merged[int(0.99 * len(merged)) - 1]   # Result is about 10 ms.

These three values differ by more than an order of magnitude. This shows that only when the collection layer reports mergeable raw statistical structures, such as histogram bucket counts, can server-side aggregated quantiles be trusted. Once precomputed quantile values are reported, the aggregated result has already lost its meaning, and this error grows with scale in multi-region, multi-shard, or multi-instance deployments. Multi-region capacity assessment and SLO attainment calculations must be built on merged bucket counts, not averaged quantiles.

6. From “A Failure at One Moment” to “It Is Getting Worse”

The core output of series-based observation is a baseline. With enough historical data, it becomes possible to establish a normal range for each metric that changes over time. This helps distinguish three situations:

• Normal fluctuations that follow seasonal patterns.
• Real degradation caused by slow drift.
• Sudden changes that have no historical basis.

Two common approaches are used to detect gradual degradation:

1. Window-based trend testing: apply linear regression within a sliding window. If the slope is significantly non-zero and continues in the same direction across multiple windows, the metric is considered to be drifting. This is suitable for one-directional degradation such as memory growth, rising connection counts, or queue backlog.
2. Cumulative sum testing: accumulate deviations between observed values and target values. This can identify small but directionally stable shifts earlier than mean-based tests, but it is more sensitive to temporary spikes and should be combined with denoising.

def has_drift(series, window: int, min_slope: float, patience: int = 3):
    """Return True when slopes in multiple consecutive windows are large and in the same direction."""
    hits = 0
    for w in sliding_windows(series, window):
        slope = linear_regression_slope(w)
        if abs(slope) >= min_slope:
            hits += 1
        else:
            hits = 0            # Reset after an interruption to avoid treating noise as a trend.
        if hits >= patience:
            return True, slope
    return False, None

The benefit of detecting gradual degradation is that it moves the response earlier: instead of “rescue when an alarm fires,” the team can adjust release cadence and capacity decisions. This is something a single-threshold mechanism cannot provide structurally.

7. Alert Design: Multi-Window Burn Rates and Tiered Response

SLO-oriented alerting should use a multi-window, multi-burn-rate structure. Short windows are responsible for quickly discovering sudden incidents, while long windows filter out noise. Different response levels should be triggered only when both short and long windows exceed their thresholds.

# Illustrative thresholds; recalibrate them according to your own SLO and traffic characteristics.
def classify(burn_1h, burn_6h, burn_24h):
    if burn_1h > 14.4:                       return "page"    # Immediate on-call page.
    if burn_6h > 6 and burn_24h > 3:         return "ticket"  # Handle during working hours.
    if burn_24h > 1:                         return "backlog" # Add to improvement backlog.
    return None

Three design principles should accompany this structure:

• Every alert must be bound to an executable response action. Rules that cannot trigger action should be downgraded to dashboard information.
• Alerts need noise-suppression mechanisms, including aggregation of alerts from the same source, maintenance-window silences, and dependency-based suppression. Otherwise, alert fatigue will render the entire system ineffective within weeks.
• Alert thresholds themselves need version control to avoid unnoticed drift during long-term operation.

8. Data Integrity: Do Not Fabricate Clarity When Data Is Missing

Probe failures, network interruptions, and process restarts create gaps in collected data. The most common mistake is filling missing values with zero.

For counter-type metrics, filling with zero can later be interpreted by difference calculations as “request volume dropped to zero,” causing abnormal error-rate denominators and incorrect alerting logic. For latency metrics, filling with zero artificially lowers quantiles.

The correct approach is to explicitly distinguish four states:

• Normal value.
• Known zero value.
• Unknown, represented as a gap.
• Unavailable, such as a failed probe.

In visualization, gaps should appear as broken lines rather than values stuck to zero. When calculating quantiles, missing data should be excluded rather than counted as zero. In addition, each collection channel should have an independent “heartbeat” metric so that missing data itself becomes an alertable event. Otherwise, the most serious observability failures often appear in the form of “the dashboard is quiet.”

9. Common Anti-Patterns

• Metric fragmentation: adding independent metrics for each requirement without unified label semantics, making cross-service comparison impossible.
• Collection without governance: no retention tiering and no process for retiring low-value metrics, causing storage and query costs to increase monotonically over time.
• Misusing quantiles during aggregation: merging statistics at the wrong layer, as described in the quantile section, producing numbers that look precise but are meaningless.
• Replacing investigation with dashboards: refreshing dashboards instead of performing root-cause analysis, with no ability to drill down at the trace level.
• Treating alerts as conclusions: treating the firing of an alert as the problem definition, without SLOs and error budgets as a unified decision standard.

10. Implementation Checklist

1. Are the four core signals present for every critical service, and can they be interpreted across dimensions?
2. Is latency collected as bucket counts, and do the bucket boundaries cover the expected quantile range?
3. Is metric label cardinality controlled, and are high-cardinality fields moved down to logs and traces?
4. Are raw, downsampled, and aggregated retention layers implemented, and do downsampling rules preserve peak values and counter-difference semantics?
5. Are all cross-instance and cross-region aggregations based on mergeable statistical structures rather than averaged quantiles?
6. Is there trend detection for one-directional degradation, and can it distinguish seasonal fluctuations from real drift?
7. Do alerts use multi-window burn rates, and does every rule have a clear response action and suppression strategy?
8. Are missing-data cases explicitly modeled, and do collection channels have independent heartbeats and failure alerts?
9. Is there a periodic retirement process for low-value metrics to avoid long-term uncontrolled growth?