Digital Asset Market Data Service: Architecture and Implementation Practices

    This article examines the technical architecture and engineering practices behind a digital asset market data service. Starting from the semantics of market snapshot fields, it covers multi-source collection and normalization, streaming aggregation, tiered storage and caching, distribution and push mechanisms, and quality monitoring and alerting. It emphasizes consistent calculation conventions, adapter-based ingestion, deduplication and out-of-order handling, coalesced delivery, and end-to-end latency visibility, providing a practical reference for market data display, quantitative trading, and risk monitoring systems.

    In digital assets, market data is a foundation for every business system. Trading terminal visualization, quantitative strategy signal calculation, and real-time risk monitoring all depend on a stable, low-latency, and highly accurate market data pipeline. Using a typical market snapshot record as an analytical sample, this article starts with field semantics and then expands layer by layer into the complete technical architecture and engineering practices of a market data service.

    1. Interpreting Market Snapshot Fields

    A standard market snapshot usually contains the following core fields: asset code, trading pair identifier, market type, last traded price, quote currency, 24-hour change amount and change percentage, open price, period high and low, market capitalization, market capitalization rank, and data update time. Understanding the source and calculation method of each field is the first step in building a market data service.

    The last price is typically the price of the most recent matched trade. The open, high, and low prices are generally aggregated over a natural day or a rolling 24-hour window. The percentage change calculation requires particular attention. The industry commonly uses the formula (last price - period open price) / period open price. For example, when the last price is 0.09062 and the open price 24 hours earlier is 0.09292, the change amount is approximately -0.0023, corresponding to a decline of about 2.47%. During development, calculation conventions must be unified; otherwise, different modules may display inconsistent percentage changes and raise user concerns.

    The market capitalization field is usually derived from circulating supply × last price. It is a derived metric rather than raw market data. Market capitalization rank depends on sorting a global asset list. It is an offline or near-real-time task and should not be calculated synchronously in a high-frequency market data pipeline.

    2. Data Collection Layer Design

    Market data originates from upstream data interfaces or matching transaction streams. The collection layer needs to address three core issues:

    1. Multi-source access and normalization. Different data sources may return inconsistent data formats, timestamp precision, and currency units. The collection layer should use a unified adapter pattern to convert external messages into an internal standard model. The following illustrative data structure can be used:

    1. {
    2. "symbol": "ASSET_USD",
    3. "marketType": "CRYPTO",
    4. "lastPrice": "0.09062",
    5. "open24h": "0.09292",
    6. "high24h": "0.09412",
    7. "low24h": "0.08922",
    8. "changePct": "-2.473",
    9. "quoteCurrency": "USD",
    10. "eventTime": "2026-11-20T16:12:55Z"
    11. }

    Price fields should be transmitted as strings or high-precision decimal types to avoid price display errors caused by floating-point precision loss.

    2. Reconnection and data gap filling. Real-time streaming push may be interrupted by network jitter. The collection layer must support automatic reconnection and, after reconnection, use snapshot interfaces to fill data gaps during the interruption, ensuring continuity of the K-line series.

    3. Deduplication and out-of-order handling. Duplicate pushes and out-of-order arrivals are difficult to avoid in distributed pipelines. Each market event should be assigned a monotonically increasing sequence number. Consumers use the sequence number to determine freshness and discard stale data.

    3. Computation and Aggregation Layer

    Raw tick data is massive, and storing and querying it directly is expensive. Therefore, an aggregation layer is needed for downsampling:

    • Real-time K-line aggregation: Perform windowed aggregation on streaming data for common periods such as 1 minute, 5 minutes, 1 hour, and 1 day, producing OHLCV structures (open, high, low, close, and volume). A streaming computation engine or an in-memory sliding window can be used.
    • Derived metric calculation: Metrics such as percentage change, amplitude, turnover rate, and market capitalization are calculated uniformly in the aggregation layer, avoiding duplicated downstream implementations and inconsistent conventions.
    • Snapshot merging: For display scenarios, maintain a merged view of the latest snapshot plus 24-hour statistics so the API layer can read it directly and keep query latency at the millisecond level.

    In engineering implementation, aggregation results should be written to both cache and persistent storage. Cache handles high-frequency read requests, while persistent storage supports historical backtracking and failure recovery.

    4. Storage and Cache Selection

    Market data has typical time-series characteristics: more writes than reads, queries by time range, and historical data that is rarely updated. The storage layer can adopt a tiered design:

    1. Hot data layer: Use in-memory cache to store the latest snapshots and recent K-lines, supporting high-concurrency polling on market list pages. Cache keys can be designed by trading pair, with appropriate expiration and active invalidation policies.
    2. Warm data layer: A time-series database is suitable for storing minute-level and hour-level K-lines and provides efficient time-range aggregation queries.
    3. Cold data layer: Historical archived data can be written to object storage with columnar compression formats to reduce costs, and loaded on demand for backtesting and audit scenarios.

    It is important to note that global sorting results such as market capitalization rank have low update frequency (minute-level is sufficient). They should be refreshed in batches by scheduled tasks rather than recalculated in real time with every market tick.

    5. Distribution and Push Mechanisms

    There are two main modes for distributing market data to end users:

    • Polling: Simple to implement and suitable for low-frequency refresh on list pages (for example, every 5 to 10 seconds). The server should support conditional requests and incremental responses to reduce bandwidth consumption.
    • Long-connection push: Push price changes to subscribers in real time through full-duplex long connections, suitable for trading detail pages. The server needs to manage subscriptions, detect connection heartbeats, and control message backlog. When the message backlog on a single connection exceeds a threshold, it should proactively downgrade to merged snapshot push to avoid memory growth.

    Push frequency should be coalesced to smooth peaks: when the same trading pair changes multiple times within a very short window, only the latest value at the end of the window is pushed. This maintains real-time user experience while significantly reducing system load.

    6. Quality Monitoring and Alerting

    If a market data service experiences data stagnation, abnormal price jumps, or calculation convention errors, it directly affects user decisions. Therefore, a monitoring system is essential:

    • Freshness monitoring: Check the difference between the snapshot update time and the current time. If the difference exceeds a threshold, trigger an alert to quickly detect collection interruptions.
    • Sanity checks: Compare the last price with the previous price and with the 24-hour high/low containment relationship to identify abnormal jumps. When the percentage change exceeds a preset range, perform manual review.
    • Pipeline metric instrumentation: Instrument collection latency, aggregation latency, and push latency separately to form an end-to-end latency dashboard.

    These monitoring capabilities can be built on general-purpose logging and monitoring/alerting systems. For example, using managed logging, monitoring and alerting, and message queue services in the cloud can reduce self-built operations and maintenance costs and allow the team to focus on market data business logic.

    7. Conclusion

    A reliable digital asset market data service is essentially an organic combination of five stages: collection and normalization, streaming aggregation, tiered storage, peak-shaved push, and end-to-end monitoring. Developers should start by unifying field semantics and calculation conventions, use the adapter pattern to shield multi-source differences, balance performance and cost with caching and tiered storage, and safeguard data quality with comprehensive monitoring and alerting. Mastering this general architecture methodology can support a wide range of upper-layer application scenarios, from market data display to quantitative trading.