Real-Time Market Data Systems: Data Collection and Metrics Computation in Practice
A practical engineering guide to building and operating a real-time market data system. Starting from a concrete market snapshot, it walks through the four core layers—ingestion, computation, serving, and observability—covering field-semantic modeling, data freshness, rolling 24-hour statistics, market-cap ranking, and validation rules. It also explains how to monitor freshness, completeness, and accuracy, presents a reference architecture organized around general cloud-service categories, and offers failure playbooks for common anomalies. Readers will gain a clear blueprint for turning raw quotes into trustworthy, queryable market data at scale.
1. Introduction: The Engineering Behind a Single Market Snapshot
Open any market data page and you typically see a structured record like this: a digital asset priced in USD, with a latest price of 0.09062, a 24-hour change of -0.0023, and a 24-hour change of -2.473%. The page also shows a 24-hour open of 0.09292, a high of 0.09412, a low of 0.08922, a market cap of roughly $90.05 million, and a market-cap rank of 170. Every number is tied to a single update timestamp: 2026-11-20 16:13:56.
There are only a dozen or so fields, yet producing them reliably and continuously requires a complete real-time data system. That system must answer at least four questions:
- Where do prices come from, and how do we keep them fresh?
- How are the 24-hour open, high, and low computed precisely?
- How are market cap and rank generated efficiently across thousands of assets?
- How do we detect anomalies and limit damage when data goes wrong?
Using “a single market snapshot” as the thread, this article breaks the journey “from raw quotes to queryable market data” into four layers—ingestion, computation, serving, and observability—and closes with a practical overall architecture. All field names and values here are illustrative only.
2. Data Model: Defining Field Semantics First
The first step in engineering practice is modeling, and the prerequisite for modeling is stating each field’s semantics clearly.
| Field category | Example value | Definition and notes |
|---|---|---|
| Symbol | Asset code | Globally unique identifier; the basis for mapping across data sources |
| Trading pair | Asset code / USD | Identifies the quote currency; values quoted in different currencies cannot be mixed directly |
| Market type | CRYPTO, etc. | Determines the collection method and refresh frequency |
| Latest price | 0.09062 | Usually the most recent trade price, or the midpoint of best bid and ask |
| 24-hour open | 0.09292 | The earliest baseline price in a rolling window, or the price at a day boundary (typically UTC 00:00) |
| 24-hour change | -0.0023 | Latest price minus the 24-hour open |
| 24-hour change % | -2.473% | Change divided by the 24-hour open, times 100% |
| 24-hour high / low | 0.09412 / 0.08922 | Extremes within the rolling window |
| Market cap | ~$90.05M | Latest price times circulating supply; unit determined by the quote currency |
| Market-cap rank | 170 | Position when all assets are sorted by market cap descending; a global statistic |
| Update time | 2026-11-20 16:13:56 | Must distinguish market time from capture time |
Three points deserve special attention.
First, verify that the definitions are self-consistent. Check the example: 0.09062 − 0.09292 = −0.0023, and −0.0023 ÷ 0.09292 × 100% ≈ −2.475%. Yet the displayed value is −2.473%, which tells us the value was computed from high-precision, unrounded intermediate values and then rounded to three decimal places. Working backward, the unrounded 24-hour open is about 0.092904, differing from the five-decimal 0.09292 by about 0.000016—well within normal rounding error.
The lesson for developers is direct: all metrics should be computed from the same set of high-precision raw values, then rounded consistently at output. If you compute change from an already-rounded price and then derive other fields from an already-rounded change, you get mismatches such as “change divided by open does not equal change %.”
Second, the “open” is not open in the literal sense. The “24-hour open” on a market page is generally the baseline price obtained by looking back 24 hours from the current moment. As time advances, both edges of the window slide forward together. If you mistakenly take the price at natural-day 00:00, you get an intraday change, which is a different definition from a 24-hour change; the two must be strictly separated as distinct fields.
Third, market cap and rank are global statistics. A single quote carries only price information; circulating supply comes from independent asset metadata, and rank is the position obtained after sorting all assets by market cap. Neither market cap nor rank can be derived from a single market record, so both must rely on periodically executed batch jobs. Numeric precision also needs to be agreed in advance: storing fractional prices as floating point tends to introduce last-digit errors after multiplication and division, so financial and market data typically uses fixed-point numbers or integers scaled to the smallest price unit.
3. Ingestion Layer: The First Guarantee of Data Freshness
The goal of the ingestion layer is to retrieve data as fresh as possible with as little resource overhead as possible. Five points matter in the design.
Cold/hot tiering. Attention across assets varies enormously, and uniform high-frequency polling wastes bandwidth and quota. Assets can be tiered by market-cap rank—for example, top 100 collected every few seconds, 100–500 every 10–30 seconds, and the rest once per minute.
Push over pull. Data sources usually offer both subscription-based incremental pushes and on-demand pulls. Where quota allows, prefer push: it is inherently more real-time and significantly reduces request volume. Keep the pull channel as a reconciliation fallback for recovering after disconnects.
Sharding by asset. Collection tasks are hash-partitioned across instances by asset, so updates for the same asset are always processed serially by the same instance. This prevents out-of-order updates from overwriting new data with stale data.
A unified time base. Internally, market time should be carried as a UTC millisecond timestamp, with the presentation layer converting to the user’s time zone. If a source only provides second-level time, pad the precision on storage and mark the source, so downstream jobs don’t misjudge freshness.
Stateless and retryable. Collection instances hold no business state and can be restarted by the scheduler after a crash. Transient failures use exponential backoff retries; consecutive failures raise an alert and automatically fail over to a backup source.
4. Computation Layer: Turning Raw Quotes into Trustworthy Metrics
The computation layer produces all derived metrics in the market snapshot, with three main challenges.
1. Boundary consistency of rolling windows
24-hour statistics require the open, high, and low within the window to use exactly the same start and end times. Two common approaches:
- Raw event-level sliding window: maintain all market events from the last 24 hours, keeping extremes with a monotonic double-ended queue for amortized O(1) time. This is the most precise, but memory scales with window length.
- Pre-aggregation with downsampling: first aggregate quotes into minute-level K-lines, then compute 24-hour statistics at minute granularity. This drastically reduces storage and computation cost, at the price of extreme-value precision being limited by the aggregation granularity.
For display-oriented market pages, minute-level aggregation is usually sufficient; for precision-sensitive scenarios such as trading and risk control, use the event-level approach.
Illustrative code:
# Rolling-window extremes: monotonic double-ended queue, amortized O(1)class RollingExtremes:def __init__(self, window_ms: int):self.window_ms = window_msself.max_q = deque() # monotonically decreasing: (market time, price)self.min_q = deque() # monotonically increasing: (market time, price)def push(self, ts: int, price: Decimal) -> None:# When the new value is not smaller than the tail, the tail can never be the maxwhile self.max_q and self.max_q[-1][1] <= price:self.max_q.pop()while self.min_q and self.min_q[-1][1] >= price:self.min_q.pop()self.max_q.append((ts, price))self.min_q.append((ts, price))self.evict(ts)def evict(self, now: int) -> None:deadline = now - self.window_mswhile self.max_q and self.max_q[0][0] < deadline:self.max_q.popleft()while self.min_q and self.min_q[0][0] < deadline:self.min_q.popleft()@propertydef high(self):return self.max_q[0][1] if self.max_q else None@propertydef low(self):return self.min_q[0][1] if self.min_q else None
2. Whole-market market-cap ranking
Pairwise comparison sorting can’t support minute-level refresh across thousands of assets. A better approach is layered maintenance:
- Each statistical period (e.g., every minute), sort all assets by market cap descending, generate a full rank table, and write it to cache;
- Single-asset updates within a period only affect market cap, and adjust the rank with local rules (“can only move up or down a limited number of positions”);
- At period end, correct accumulated error with a full sort.
Also set an upper bound on the worst-case rank shift. If a correction exceeds that bound, adjacent assets have nearly identical market caps, and an immediate full sort should be triggered.
3. Anomaly interception
When a source produces dirty data, derived metrics amplify it quickly, so validation rules must run before computation:
- The latest price must be greater than 0 and fall within a reasonable range;
- When the change between two consecutive quotes exceeds a set threshold, mark it as suspicious and hold publication;
- The latest price must lie between the window high and low;
- Change and change % must be derived from the same baseline, so cross-validation always holds.
Intercepted data goes to a separate anomaly channel for manual review.
5. Serving Layer: Balancing Query Performance and Data Consistency
The serving layer provides query capabilities for pages, clients, and open APIs. The core strategy is “cache-first, read/write separation.”
- Multi-level caching: a local cache holds millisecond-level hot data and serves most reads; a distributed cache holds the full snapshot for sharing across instances; a database holds recent history to support K-line and lookback queries.
- Cache invalidation: freshness is judged by the market timestamp. Each cache entry records the market time its data corresponds to; on read, entries beyond the tolerated latency are treated as expired and trigger a refill.
- Bulk queries: a batch interface returns the latest data for many assets in one round trip; market pages usually care about only a few fields such as change and change %, so the returned content can be trimmed as needed.
- Unified output: all metrics are rounded consistently at the exit and carry the data time and definition version, making upstream discrepancy analysis easier.
6. Observability: Making Data-Quality Issues Discoverable and Locatable
The most dangerous failure in a market data system is not a crash, but “the page still displays, yet the data has long been stale.” An observability system should cover at least four metric classes.
- Freshness: the difference between the latest quote’s market time and the current time, aggregated by asset as P50 and P99—the most core health metric of a market data system.
- Completeness: the missing rate and parse-failure rate of each field. For example, if a source stops returning the high-price field, the aggregate result changes silently, and only completeness monitoring can catch it.
- Accuracy: cross-validate derived metrics, e.g., verify that “change = latest price − open” holds within rounding precision, and periodically sample-compare against a backup source.
- Pipeline health: end-to-end latency from ingestion to computation and from computation to serving, plus backlog at each stage.
Also configure tiered alerts: sustained freshness breaches, sudden spikes in the anomaly ratio, and backup-source failovers trigger immediate alerts; slow-trending issues such as gradual completeness degradation go into a daily report. Centralize full-pipeline logs through a logging service, and pair them with monitoring alerts for problem localization and notification.
7. Overall Architecture: Organized by General Cloud-Product Categories
Combining all the layers, a typical real-time market data system can be organized as follows:
[Market sources] --(subscription push / scheduled pull)--> [Ingestion service cluster]| validation / normalizationv[Message queue (partitioned by asset)]/ \v v[Real-time computation] [Data archiving]latest / 24h stats / extremes (object storage)|v[Market data store (time-series DB / cache)]|v[Batch jobs: market-cap rank / minute pre-aggregation / daily reconciliation]|v[Query service] --> [Market pages / clients / open APIs]Cross-cutting: logging service, monitoring & alerting, container platform (autoscaling & self-healing)
Component responsibilities:
- The message queue partitions by asset to guarantee per-asset ordering and to decouple ingestion from computation, smoothing out peaks;
- Raw market data is archived to object storage for easy replay/recomputation and incident review;
- A time-series database or cache supports latest-price and history queries;
- Batch jobs are driven by a scheduler, periodically refreshing market-cap rankings and pre-aggregation results;
- The logging service and monitoring/alerting span the full pipeline;
- Ingestion and computation services run on a container platform and autoscale based on message backlog.
8. Failure Playbooks: Common Anomalies Seen from the Example Snapshot
Taking the example snapshot, we can reason through several typical failures and their remedies:
- The latest price stays stuck at 0.09062 for a long time: usually a source-side data freeze. Freshness monitoring raises an alert within seconds, and the query API returns the data together with a “data delayed” flag to prevent users from misjudging market state.
- Abnormal jump in the change %: usually the 24-hour open window is wrongly reset—for example, the window is cleared after the ingestion service restarts, so the baseline price becomes the first price after restart. The fix is to externalize the window state so the service can rebuild it by market time after a restart.
- Market cap and rank stay unchanged for a long time: the batch ranking job didn’t run or failed. This is detectable early by monitoring “the difference between the rank update time and the current time.”
These scenarios show that a market data system’s reliability depends not only on the main pipeline, but on how complete the monitoring and contingency plans around it are.
9. Conclusion
A seemingly simple market snapshot is actually the joint result of a data model, stream computation, batch aggregation, caching, and an observability system. For developers, the practical path starts with a minimal closed loop: single-source ingestion, minute-level window aggregation, a batch job for market-cap ranking, and a cached query API—then gradually add multi-source redundancy, anomaly interception, and full-pipeline monitoring.
Always keep three principles in mind:
- Define the metrics first, then write code;
- Use fixed-point or integer types for precision;
- Freshness, like accuracy, is a core metric that must be measured.
If a system can steadily and accurately answer “what is the latest price, what is the 24-hour change, what is the rank, and is the data fresh,” it has fulfilled its most important mission.
