Real-Time Market Data System Architecture: Design and Engineering Practices

    A practical guide to building an industrial-grade real-time market data system. It walks through the full data pipeline, from normalized ingestion and real-time stream computing to layered storage and low-latency display, and shows how to balance semantic accuracy, low latency, high throughput, and consistency. You will learn how to define quote fields, normalize upstream feeds, compute derived metrics such as price change and market-cap ranking, choose storage for hot and historical data, and enforce reliability, security, and compliance across the pipeline.

    1. Background of Market Data Systems

    In digital assets, securities trading, and various real-time bidding scenarios, the market data system is the foundational data infrastructure that supports upper-layer applications. A typical market data snapshot usually contains fields such as symbol, trading pair, market type, last traded price, quote currency, 24-hour price change and change rate, open price, high price, low price, market cap, market-cap rank, and update time. These fields look simple, but they only become usable information after high-frequency collection, real-time computation, reliable storage, and low-latency distribution.

    The structure of a real snapshot looks roughly like this:

    1. {
    2. "symbol": "A",
    3. "pair": "AUSDT",
    4. "market_type": "CRYPTO",
    5. "last_price": 0.06614,
    6. "currency": "USD",
    7. "change_24h": -0.002,
    8. "change_rate_24h": -2.933,
    9. "open_price": 0.06814,
    10. "high_price": 0.06894,
    11. "low_price": 0.06494,
    12. "market_cap": 110000000,
    13. "rank": 154,
    14. "update_time": "2026-11-20T16:09:54"
    15. }

    Building an industrial-grade market data system means balancing data semantic accuracy, processing timeliness, system throughput, and result consistency. The sections below follow the data flow layer by layer.

    2. Core Fields and Semantics of Market Data

    Understanding field semantics is the prerequisite for system design. Market data falls into three categories:

    • Price fields: last price, open price, high price, and low price. Together they describe the price fluctuation range within a statistical period. Price change and change rate are derived fields, typically computed from the last price and a reference price (such as the price 24 hours ago or the open price).
    • Metric fields: market cap, market-cap rank, and trading volume. Market cap generally equals circulating supply multiplied by the current price; rank depends on a cross-market comparison and is the result of an aggregation.
    • Metadata fields: symbol, trading pair, market type, quote currency, and update time. These fields identify and route data, ensuring it is not confused across multiple markets and currencies.

    Derived fields (price change, change rate, and rank) should not be collected as independent inputs; instead, they should be computed in real time from the raw data to avoid inconsistencies across multiple sources.

    3. Data Collection and Ingestion Layer Design

    The core goal of the collection layer is “stable ingestion and orderly normalization.” Common problems include uneven push frequency from data sources, inconsistent field naming, and messy time-zone handling in timestamps.

    A unified data model comes first. Regardless of whether upstream data arrives through a push stream or a polling interface, it should first be normalized into an internal structure:

    1. from dataclasses import dataclass
    2. from datetime import datetime
    3. @dataclass
    4. class Quote:
    5. symbol: str
    6. pair: str
    7. market_type: str
    8. last_price: float
    9. open_price: float
    10. high_price: float
    11. low_price: float
    12. currency: str
    13. update_time: datetime
    14. @property
    15. def change(self) -> float:
    16. return self.last_price - self.open_price
    17. @property
    18. def change_rate(self) -> float:
    19. if self.open_price == 0:
    20. return 0.0
    21. return self.change / self.open_price * 100

    For ingestion strategy, the push model (long-lived connections or message queue subscriptions) suits high-frequency scenarios and significantly reduces latency; the polling model is simple to implement but requires a reasonable polling frequency and deduplication. Production environments often combine the two: push as the primary path with polling as a fallback, preventing data gaps when connections drop.

    4. Real-Time Stream Computing

    Raw market data must go through real-time computation to produce derived metrics. The core abstraction of a stream computing framework is the “event stream,” where each market data change enters the processing pipeline as an event.

    Key computations include:

    • Change calculation: subtract the open price or a reference price from the last price.
    • Range aggregation: maintain the high and low price within each statistical window, which can be updated efficiently with a sliding-window structure.
    • Rank calculation: market-cap ranking requires sorting across the entire market and is a global aggregation. When the data volume is large, a strategy of bucketing plus approximation with periodic exact recomputation can trade off accuracy against performance.
    1. class WindowAggregator:
    2. def __init__(self):
    3. self.high = float("-inf")
    4. self.low = float("inf")
    5. def update(self, price: float):
    6. if price > self.high:
    7. self.high = price
    8. if price < self.low:
    9. self.low = price

    Watermarks and late data are the hard parts of stream processing. Network jitter can cause market data to arrive out of order, so a watermark mechanism is needed to tolerate a bounded range of lateness, along with defined rules for when windows trigger and close, avoiding prolonged memory retention.

    5. Storage Layer Selection

    Storage choice depends on the access pattern:

    • Latest snapshot: requires extremely low read/write latency and is best placed in an in-memory key-value store, keyed by symbol, so the market dashboard can refresh in milliseconds.
    • Historical series: stored along a timeline and used to render candlestick charts and compute long-term trends. A time-series database or columnar store is more suitable, natively supporting range scans over time and high compression ratios.
    • Hot/cold tiering: keep recent data in hot storage and archive historical data to cold storage such as object storage, balancing cost and query efficiency.

    Whatever storage is used, primary keys and indexes should be designed around data semantics. A common approach is to organize by “market + trading pair + time,” which supports locality-friendly queries.

    6. Visualization and Presentation Layer

    The ultimate value of market data lies in readable presentation. The presentation layer typically needs:

    • Real-time number dashboards: display core metrics such as last price and change rate, using color semantics to distinguish gains from losses.
    • Price curves and candlestick charts: rendered from historical series; the frontend can use delta updates (pushing only the changed parts) to reduce bandwidth and rendering pressure.
    • Push mechanism: push incremental updates to clients over long-lived connections, avoiding the latency and resource waste of polling.
    1. // Delta update example: only re-render the DOM when the price changes
    2. socket.on("quote", (quote) => {
    3. const el = document.getElementById(quote.symbol);
    4. if (el && el.dataset.price !== String(quote.last_price)) {
    5. el.textContent = quote.last_price.toFixed(5);
    6. el.dataset.price = quote.last_price;
    7. el.classList.toggle("up", quote.change_rate >= 0);
    8. }
    9. });

    7. Consistency, Reliability, and Security Compliance

    Data consistency: derived fields must be computed from the same original snapshot, avoiding mismatches between how price change and change rate are calculated. The whole pipeline should carry a unified update timestamp so the frontend can judge data freshness.

    System reliability: the collection, computation, and storage stages should support resumable transfer and idempotent processing. A message queue guarantees “at-least-once” delivery, which combined with deduplication achieves an effective “exactly-once” result. Set up monitoring and alerting on critical paths so that when data latency or loss exceeds a threshold, an alert is sent promptly.

    Security and compliance: although market data is inherently public information, the system must still guard against malicious high-frequency scraping, unauthorized access, and data tampering. Apply rate limiting, authentication, and auditing to interfaces, and enable encryption on transport links to ensure that data comes from trusted sources and goes to controlled destinations. For data with financial attributes, the presentation should clearly label the update time and a data description to avoid misleading users.

    8. Conclusion

    Building a real-time market data system is essentially about turning scattered, heterogeneous, high-frequency raw market data into accurate, timely, decision-ready information. Its technical backbone can be summarized as: normalized ingestion → stream computing → layered storage → low-latency display → end-to-end reliability and security. Along this path, the accuracy of field semantics is the foundation, the timeliness of stream processing is the core, the coordination of storage and display is the value outlet, and consistency and security compliance are the prerequisites for long-term stable operation. By mastering these general engineering methods, developers can confidently handle the full challenge from raw data to trusted information in any high-concurrency, real-time data scenario.