Digital Asset Market Data Service Design: Architecture, Precision, and Real-Time Delivery
A production-grade market data service for digital assets has to be accurate, real-time, and always available. Starting from a single ticker snapshot, this article lays out a complete design across data collection, processing, storage, push delivery, and monitoring. It covers unifying field semantics, high-precision decimal calculations, rolling 24-hour windows, multi-source fusion, invariant validation, hot/warm/cold storage layering, snapshot-plus-increment real-time push, and observability metrics. Readers will learn how to define where the numbers come from, how precise they are, when they refresh, and what to do when errors appear, along with common pitfalls and a phased rollout roadmap.
1. Introduction: Starting from a Single Ticker Record
In digital-asset applications, market data is the shared foundation for trading displays, portfolio monitoring, quantitative strategies, and risk alerting. These applications place three core demands on market data: accuracy (the numbers must hold up to verification), timeliness (updates should reach users as fast as possible), and availability (the service must keep running even when one data source fails). The example record below is taken from a ticker snapshot endpoint. For clarity, the asset symbol has been anonymized:
Symbol: GLMPair: GLMUSDTMarket type: CRYPTOLast price: 0.09022 USD24h change: -0.0027 USD24h change (%): -2.903%Open / High / Low: 0.09292 / 0.09412 / 0.08922 USDMarket cap: USD 90.05 millionMarket cap rank: 170Updated: 2026-11-20 16:07:40
This record carries a surprising amount of information: it includes market metrics such as last price, change percentage, and 24-hour high and low, plus derived metrics like market cap and rank, along with metadata such as the trading pair, quote currency, and timestamp.
From an engineering standpoint, many production incidents trace back to details that look trivial:
- Is the change percentage computed from the display value or from the internal high-precision value?
- Does market cap use circulating supply or total supply?
- When does the ranking refresh, and how stale is it from the client’s point of view?
- Is the timestamp the exchange event time, the data-generation time, or the system write time?
- How do the collector and the service behave when an asset’s price moves sharply?
This article addresses these questions and presents a complete design spanning data collection, processing, storage, push delivery, and monitoring.
2. Overall Architecture: Four Layers Plus a Monitoring Backbone
A production-grade market data service can be divided into four layers:
- Collection layer: connects to multiple data sources and handles snapshot pulls and incremental subscriptions;
- Compute layer: performs normalization, filtering, rolling statistics, multi-source fusion, and consistency checks;
- Storage layer: separately holds the latest snapshot, periodic K-lines, and raw detail, organized by access frequency;
- Service layer: exposes data to clients through two channels—HTTP endpoints and long-lived push connections.
Running through all four layers is a monitoring backbone responsible for observing latency, completeness, and accuracy, and for triggering alerts when anomalies occur.
The layers should be decoupled through a message queue. The compute layer can then scale independently; when the collector experiences a brief outage, messages accumulate in the queue and are replayed in order after recovery; and every layer achieves idempotent processing via a business-level unique key, so retries never cause duplicate data or error accumulation.
2.1 Data Source Integration
Common data sources fall into three categories:
- Market data endpoints: snapshots for a single or multiple trading pairs;
- Incremental subscriptions: trade-by-trade executions and order-book changes pushed over a long-lived connection;
- Derived-metric endpoints: computed results such as market cap, circulating supply, and rank.
Different sources vary widely in push frequency, precision, and history depth. The collection layer should isolate these differences inside per-source adapters and emit a unified standard structure upstream:
{"source": "venue_a","symbol": "ASSET_A/USD","market_type": "CRYPTO","event_time": "2026-11-20T16:07:40Z","ingest_time": "2026-11-20T16:07:41Z","last": "0.090223","open_24h": "0.092927","seq": 10086234}
Key design points:
- Unified field semantics: a source’s “last price” may come from the last trade, the order-book midpoint, or an index price; on ingest it must be mapped to a single agreed definition. Data whose semantics don’t match should be rejected rather than filled in with a default value.
- Amounts carried as strings: monetary fields such as price and change must be transmitted as strings and parsed as high-precision fixed-point decimals, never held as floating-point numbers.
- Two timestamps are mandatory:
event_timemarks when the data was produced and is used for out-of-order detection and window calculations;ingest_timemarks when the data arrived locally and is used to measure collection latency. Confusing the two makes it extremely hard to diagnose data-lag issues later.
2.2 Choosing a Collection Mode
Snapshot pulls and incremental subscriptions are not mutually exclusive—they are complementary. A common production approach is:
- On startup, pull one full snapshot as a baseline;
- Then subscribe to the incremental stream for low-latency updates;
- Periodically pull a snapshot to reconcile, correcting missed messages, out-of-order data, and sequence gaps.
Relying only on the incremental stream means data during a disconnection can never be recovered; relying only on snapshots means the update frequency is bounded by endpoint rate limits.
2.3 Robustness at the Collector
- Adaptive rate limiting: token-bucket rate limiting plus exponential-backoff retries; when a limit is hit, lower the pull frequency to avoid a cascading failure.
- Timeouts and circuit breaking: set connect and read timeouts for every request; after consecutive failures, temporarily isolate the source and probe for recovery in a half-open state at regular intervals.
- Sequence-number deduplication: use (source, symbol, sequence number) as the message unique key to guarantee exactly-once results under at-least-once delivery.
- Local disk buffering: the collector keeps a short-term local ring buffer. When the upstream is briefly unavailable, the compute layer can replay from the buffer after recovery without a full re-pull.
3. Field Semantics and Calculation Rules: The First Line of Defense for Precision
The normalization layer must first define each metric’s meaning and calculation method. It’s best to express these rules as configuration rather than scattering them across code:
| Metric | Definition and rule | Common mistake |
|---|---|---|
| Last price | The latest trade price for the chosen market type, with the quote currency stated explicitly | Mixing rules across different market types |
| 24h open | The price of a trade whose timestamp is no later than the start of the 24-hour window ending now | Confusing it with the UTC-midnight open (daily open) |
| High / Low | The highest and lowest trade prices within the same rolling 24-hour window | Using the first trade after the window start |
| Change / Change % | Last price minus open; then divided by the open | Computing from rounded display values |
| Market cap | Last price times circulating supply (not total supply), converted to the quote currency | Mixing total supply, or summing across currencies directly |
| Market cap rank | Market caps ordered descending under a single consistent rule | Inconsistent source rankings without stating the rule |
| Timestamp | Stored in UTC, displayed in the client’s time zone | Storing only local time without a time zone |
The example data illustrates several key details.
3.1 Rolling 24-Hour Windows and Precision Loss
The 24-hour window here is a rolling window, not a calendar day. The open price is the last trade before the window start; the high and low cover all trades within the window. So the same pair can have a new “daily open” repeatedly throughout the day.
Precision problems are more subtle. Computing directly from display values:
(0.09022 - 0.09292) / 0.09292 ≈ -2.906%
The page displays -2.903%. Both results are rounded to three decimal places, yet they differ—which shows that the display values have already lost precision during rounding. On the server, two rules should apply:
- Change and change percentage are always computed from high-precision raw values;
- Rounding happens only at final output, with amounts and percentages rounded independently.
You should also configure, per pair, the number of decimal places for the price, the change amount, and the change percentage. For a low-priced asset like the 0.09022 in the example, the price keeps 5 decimal places and the change keeps 4; if the frontend uniformly formats to 2 decimal places, the change would render as 0.00, and users would immediately assume the data is broken.
3.2 Precision and Anomalies in Low-Priced Assets
When the last price is far below the high (about 4.1% lower in the example), there are two possibilities:
- A genuine rapid decline occurred within the window;
- A source switch changed the rule, or an individual source returned dirty data.
The normalization layer should not directly “correct” values; instead, it should run three levels of checks:
- Record the raw value and its source;
- Flag suspicious data using cross-source comparison and statistical thresholds;
- Hand anomalous samples to a human reviewer or a rules engine.
It’s worth stressing: filtering anomalies and preserving raw data must both happen. Overly aggressive filtering hides real market moves, while passing data through with no validation lets errors contaminate downstream strategies directly.
3.3 Market Cap Calculation
Market cap equals the last price times circulating supply, where supply typically comes from the asset issuer’s disclosure or third-party statistics. Different sources disagree on what counts as “circulating” (whether to include locked tokens, team allocations, foundation reserves, and so on). Therefore:
- The same asset may show different market caps across sources, so displays must state the data source and the counting rule;
- Circulating supply should be collected and versioned independently, not coupled to price in the same update;
- Only when circulating supply is missing or stale should the system fall back to the most recent valid supply times the current price—and mark it explicitly as an estimate.
4. Multi-Source Fusion and Consistency Checks
A single source can’t guarantee accuracy, so multi-source fusion is common practice. Before fusing, assess each source’s health—update frequency, sequence completeness, deviation from the median price, and historical accuracy—then fuse across the healthy sources.
4.1 Fusion Strategies
- Volume weighting: weight by the volume each source contributes, which stays closer to real trading;
- Median: more robust when there are few sources of uneven quality;
- Threshold rejection: a source deviating from the median beyond a set threshold is excluded from fusion and counted toward that source’s anomaly total.
The fusion result should retain three pieces of information: the fused price, the list of participating sources, and a data-quality score. The downstream display may show only one price, but risk-control and strategy modules need to know how trustworthy that price is.
4.2 Invariant Checks
Run a set of “invariants” on the normalized data; samples that fail go to a dead-letter queue for manual investigation rather than being silently dropped:
from decimal import Decimaldef check_invariants(snap) -> list[str]:"""Return a list of violation messages; an empty list means the check passed."""errors = []if not (Decimal(0) < snap.open_24h <= snap.high_24h):errors.append("open_24h must be in the range (0, high_24h]")if not (Decimal(0) < snap.low_24h <= snap.high_24h):errors.append("low_24h must be in the range (0, high_24h]")if not (snap.low_24h <= snap.last <= snap.high_24h):errors.append("last must be in the range [low_24h, high_24h]")diff = snap.last - snap.open_24hif diff != snap.change_24h:errors.append("change_24h is inconsistent with last - open_24h")return errors
Beyond passing the checks, you also need to monitor “sequence freshness”—the gap between each pair’s latest timestamp and the current time. A pair that hasn’t updated for a long time may have been halted or delisted and should be flagged separately, rather than continuing to display a stale value.
5. Storage Design: Layering by Access Pattern
Market data shows clear hot/cold tiers: last-price lookups demand millisecond responses; recent K-lines serve chart rendering and strategy replay; historical detail is mainly used for backtesting and auditing. A three-tier storage design is therefore recommended:
| Tier | Contents | Typical access pattern | Technology form |
|---|---|---|---|
| Hot | Latest snapshots, most recent K-lines | High-concurrency point and range queries | In-memory cache + local cache, with TTL and version control |
| Warm | Minute-, hour-, and day-level K-lines, partitioned by pair | Range queries, aggregation queries | Time-series or analytical database |
| Cold | Trade-by-trade and snapshot detail, raw message archive | Low-frequency scans, backfill and compliance audits | Columnar files on object storage, partitioned by day |
5.1 Key Table Designs
-- Minute K-line (warm tier): primary key includes symbol / timeframe / bucketCREATE TABLE kline_1m (symbol VARCHAR(32) NOT NULL,bucket TIMESTAMP NOT NULL, -- UTC-aligned start of the minute bucketopen DECIMAL(24, 10) NOT NULL,high DECIMAL(24, 10) NOT NULL,low DECIMAL(24, 10) NOT NULL,close DECIMAL(24, 10) NOT NULL,base_volume DECIMAL(28, 10) NOT NULL,quote_volume DECIMAL(28, 10) NOT NULL,trade_count BIGINT NOT NULL,updated_at TIMESTAMP NOT NULL,PRIMARY KEY (symbol, bucket));-- Ticker snapshot (hot/warm tier): unique key is source + symbol + market_typeCREATE TABLE ticker_snapshot (source VARCHAR(32) NOT NULL,symbol VARCHAR(32) NOT NULL,market_type VARCHAR(16) NOT NULL, -- market type such as CRYPTOquote_currency VARCHAR(10) NOT NULL,last_price DECIMAL(24, 10) NOT NULL,open_24h DECIMAL(24, 10) NOT NULL,high_24h DECIMAL(24, 10) NOT NULL,low_24h DECIMAL(24, 10) NOT NULL,change_24h DECIMAL(24, 10) NOT NULL,change_pct_24h DECIMAL(12, 6) NOT NULL,event_time TIMESTAMP NOT NULL,ingest_time TIMESTAMP NOT NULL,version BIGINT NOT NULL,PRIMARY KEY (source, symbol, market_type));
Additional notes:
versionincreases monotonically; the writer uses a conditional update (WHERE version < new_version) to prevent out-of-order overwrites;DECIMALprecision is set according to the price magnitude—a common choice is 20 integer digits and 10 to 18 fractional digits;- The warm tier is indexed jointly on (symbol, time range) to match the query pattern where charts pull a range of K-lines per pair.
5.2 Hot/Cold Separation and Lifecycle Management
- Downsampling: aggregate minute K-lines into hourly and daily K-lines; the aggregation rules (open from the first, close from the last, high/low from the extremes, volume summed) must be fixed so different pipelines don’t produce different results.
- Lower-precision storage: the cold tier can store early history at lower precision to cut storage cost, but the reduction strategy must be recorded so backtest results stay interpretable.
- Partitioning and archiving: the warm tier is partitioned by month, the cold tier by day; warm data past its retention period is archived to the cold tier and cleaned up automatically by a lifecycle policy.
- Multi-availability-zone deployment: core data should be deployed across at least two availability zones to avoid a single point of failure.
On the cloud, these tiers can map onto general-purpose offerings such as caching services, time-series databases, and object storage. Platforms such as Baidu AI Cloud typically provide these general-purpose categories; refer to the official documentation for specific specs and pricing.
6. Real-Time Push and API Services
6.1 Push Model
Market updates are a high-frequency, small-message scenario, so long-lived push is far more efficient than client polling. The design must ensure three things:
- Clients subscribe to a (pair, market type) combination, not the whole market;
- The server maintains a subscription table and pushes only the fields a subscriber cares about when data changes;
- Multiple updates for the same pair within a very short window are coalesced (for example, pushing at most once every 100 ms), balancing latency and throughput.
import asyncioasync def push_worker(conn, last_sent: dict[str, float]):"""Coalesce consecutive updates for the same pair and push on a fixed cadence."""while True:batch = await conn.queue.get() # block waiting for this cycle's updatestry:# Drain the queue without blocking, merging this cycle's updates per symbolwhile True:try:batch.append(conn.queue.get_nowait())except asyncio.QueueEmpty:breaklatest: dict[str, dict] = {}for msg in batch:latest[msg["symbol"]] = msg # keep only the latest message per symbolfor symbol, msg in latest.items():if msg["last"] == last_sent.get(symbol):continue # price unchanged, skip the duplicate pushlast_sent[symbol] = msg["last"]await conn.send_json(msg)finally:conn.queue.task_done()
6.2 Reliability Design
- Snapshot plus increment: after a client connects, it first pulls a full snapshot over HTTP, then receives incremental updates over the long-lived connection, avoiding the “push before baseline” stitching problem.
- Heartbeat and timeout: send a heartbeat at a fixed interval; after several consecutive missed heartbeats, the client proactively reconnects.
- Reconnect backoff: use exponential backoff with random jitter to prevent a flood of simultaneous reconnects when the server restarts.
- Rate limiting and degradation: limit per-connection rate; on server anomalies, return the most recent valid snapshot with a timestamp so the client can distinguish real-time data from degraded data.
6.3 HTTP Endpoint Caching
- The last-price endpoint returns
Cache-Control: no-storeor a very shortmax-age; slowly changing data like K-lines can return an ETag, and when the client sendsIf-None-Match, respond with 304 to save bandwidth; - Monetary fields returned externally are uniformly strings, formatted by the client as needed;
- State units and currency explicitly (for example,
quote_currencyis USD) to avoid ambiguity from the frontend displaying in a local currency.
7. Monitoring, Alerting, and Quality Assurance
Observability metrics for a market data service fall into five categories:
- Latency: collection lag (
ingest_time − event_time), push latency, and endpoint P99 latency; - Completeness: per-pair, per-minute sequence-gap count and heartbeat-loss rate;
- Accuracy: cross-source price deviation, invariant-check failure rate, and snapshot-versus-increment reconciliation difference;
- Availability: number of online sources, connection failure rate, and circuit-breaker trips;
- Capacity: message-queue backlog and storage write rate.
Alerts should be tiered by blast radius:
- When a single pair’s sequence is interrupted, flag only that pair’s status;
- When multiple sources deviate from the median beyond the threshold at once, trigger a data-quality alert;
- When push latency exceeds the threshold, trigger a pipeline alert.
In addition, establish a snapshot-retention mechanism. Market data is external data that cannot be rolled back; once a calculation is wrong, it contaminates every downstream conclusion. Archiving immutable, market-wide snapshots of key moments on a fixed schedule both supports post-mortems and satisfies audit requirements.
8. Common Pitfalls
- Representing prices as floating-point numbers: floating-point precision is relative, so change comparisons on low-priced assets can produce results that look equal but aren’t; use fixed-point decimals.
- Treating display values as computation values: computing derived metrics after rounding introduces visible deviations, exactly as in the example at the start of this article.
- Treating market cap as circulating market cap: using total supply overstates the actual circulating scale.
- Ignoring time zones: storing in UTC and displaying in the client’s time zone is a baseline requirement for cross-region services.
- Depending on a single data source: with no degradation path when a source fails, the service breaks at the critical moment.
- Monitoring only endpoint availability: an endpoint returning 200 doesn’t mean the data is correct—data-quality metrics must be monitored alongside availability.
9. Rollout Roadmap and Summary
A three-phase rollout is recommended:
- Getting started: integrate one or two data sources, complete field normalization, fixed-point decimal computation, and invariant checks, and expose last price and K-lines over HTTP;
- Scaling up: add incremental subscriptions and multi-source fusion, plus a caching layer, time-series storage, and a push channel, and establish latency and completeness monitoring;
- At scale: complete hot/cold tiering and lifecycle management, multi-availability-zone disaster recovery, snapshot retention and audit capabilities, and provide data views at different precision levels to different business teams.
The difficulty of a market data service lies not in calling any single endpoint, but in clearly defining four questions—where the numbers come from, how precise they are, when they refresh, and what to do when errors occur—and backing them with engineering at every layer. Starting from one concrete ticker record, this article has walked through the key decisions across collection, computation, storage, push, and monitoring, and it’s meant to offer readers building a market data pipeline a reference they can put into practice directly.
