Real-Time Market Data Systems: Ingestion, Processing, and Visualization Practices
This article presents a practical engineering approach for building real-time market data systems, covering data modeling, push and polling ingestion, validation and derived metrics, layered storage and caching, low-latency visualization, monitoring, and compliance considerations. It provides a reusable architecture for turning continuously changing market values into trustworthy, usable, and visualizable data services.
Real-Time Market Data Systems: Ingestion, Processing, and Visualization Practices
1. Background and Problem Definition
Market data is a typical example of high-frequency, time-series, and strongly consistent data. Whether the underlying assets are digital assets, securities, or other tradable instruments, market data APIs usually return a set of structured fields: trading pair symbol, latest traded price, settlement currency, 24-hour price change and percentage change, opening price, highest price, lowest price, market capitalization, market-cap ranking, and data update time.
These fields may look simple, but building a service that supports real-time refresh, multi-symbol subscription, and historical review is difficult in three dimensions:
- Real-time performance: Data must be updated at second-level or finer granularity, while ensuring that the values displayed on the frontend remain consistent with backend calculations.
- Consistency: The same instrument should not show obvious jumps across different nodes or cache layers.
- Scalability: The number of instruments, subscribers, and write operations can grow linearly or even superlinearly as the business expands.
Taking the market data of a trading pair within a specific market category as a starting point, this article abstracts a general technical solution and discusses how to turn scattered market values into a trustworthy, usable, and visualizable data service.
2. Market Data Model
Before implementation, it is important to define field semantics because they are the foundation for all subsequent calculations and displays. A market record can be modeled as follows:
from dataclasses import dataclassfrom datetime import datetime@dataclassclass MarketSnapshot:symbol: str # Trading pair identifier, e.g., "XXXUSDT"market_type: str # Market category, e.g., "CRYPTO"base_currency: str # Settlement currency, e.g., "USD"last_price: float # Latest traded priceopen_price: float # Opening price for the periodhigh_price: float # Highest price for the periodlow_price: float # Lowest price for the periodchange_amount: float # 24-hour price changechange_percent: float# 24-hour percentage changemarket_cap: float # Market capitalizationrank: int # Market-cap rankingupdated_at: datetime # Market data update timestamp
Several modeling principles should be emphasized:
- Precision. Price fields should use fixed-point types, such as
Decimalor integers representing the smallest price unit, to avoid floating-point errors amplifying during addition and subtraction. Price change amount and percentage should be derived from the latest price and a baseline price. Do not calculate them separately on multiple sides and then write them back, as this easily leads to inconsistency. - Derived fields. Price change amount, price change percentage, and periodic high/low values are derived data. They should be calculated once in the ingestion layer and stored for downstream consumers to read directly, rather than recalculated by every consumer.
- Time semantics.
updated_atrepresents the timestamp of the market data, not the time the record was inserted into the database. The two must be recorded separately: the former is used to determine data freshness, while the latter is used to investigate pipeline latency.
3. Data Ingestion Layer: Push or Poll?
Market data sources usually provide two ingestion modes: active push over a long-lived connection and passive polling through short requests. Each has trade-offs:
- Push mode: The server sends incremental updates or snapshots. It has low latency and lower resource usage, making it suitable for high-frequency instruments. However, it requires handling reconnection, heartbeat keep-alive, out-of-order messages, and duplicate messages.
- Polling mode: The client pulls data at fixed intervals. It is simple to implement and stateless, but it faces the dilemma of either wasting resources through frequent requests or introducing high latency through sparse requests.
In engineering practice, a hybrid strategy is common: use long-lived connections for high-frequency and important instruments, and use polling with exponential backoff as a fallback for low-frequency instruments. Regardless of the mode, the ingestion layer should encapsulate a data source adapter that isolates differences between sources and exposes a unified standardized event stream externally.
import time, randomdef poll_with_backoff(fetch_once, max_interval=60, base=1):interval = basewhile True:try:data = fetch_once()interval = base # Reset backoff on successyield dataexcept Exception:interval = min(interval * 2, max_interval)yield None# Add jitter to avoid simultaneous retries causing a thundering herdtime.sleep(interval + random.uniform(0, interval * 0.1))
For long-lived connections, heartbeat detection and automatic reconnection are essential. A compensation mechanism should also be designed for data gaps during disconnection, ensuring that the market data sequence is not permanently missing because of brief instability.
4. Data Processing and Metric Calculation
After data enters the system, a calculation layer is responsible for cleaning, validation, and derived calculations:
- Validation. Remove obviously abnormal values, such as prices that are zero or negative, percentage changes beyond reasonable thresholds, or timestamps that are too old, to prevent dirty data from polluting downstream systems.
- Derived calculation. Calculate price change and percentage change based on the opening price:
def derive_change(last_price, open_price):if open_price <= 0:return 0.0, 0.0change_amount = last_price - open_pricechange_percent = change_amount / open_price * 100return round(change_amount, 6), round(change_percent, 3)
- Market capitalization and ranking. Market capitalization equals the latest price multiplied by circulating supply. Ranking requires sorting all instruments by market capitalization. This is a task with heavy recomputation but relatively slow changes. A combination of incremental updates and scheduled full recalculation is suitable: update only changed instruments daily, and periodically trigger a full re-ranking to correct accumulated deviations.
5. Storage and Caching Strategy
Market data storage should be layered to balance write throughput and read latency:
- Hot data, such as the latest snapshot: Store it in an in-memory cache, with the instrument identifier as the key and the normalized latest snapshot as the value. Frontend queries can go through the cache, keeping response time at the millisecond level.
- Warm data, such as minute-level candlesticks and recent series: Write it to a time-series database or time-series-oriented storage, leveraging its compression and aggregation capabilities for the time dimension to support chart review.
- Cold data, such as the complete historical dataset: Archive it to low-cost media such as object storage, partitioned by time, for offline analysis and compliance auditing.
The cache layer also needs attention to consistency. When the calculation layer updates a snapshot, it can update the cache first and persist asynchronously afterward. Each record should include a version number or timestamp to prevent old data from overwriting new data. A message queue can be used to decouple the ingestion layer from the calculation layer, helping with peak shaving, valley filling, and horizontal scaling.
6. Visualization and Real-Time Refresh
The core goals of frontend display are low latency, no flickering, and trustworthy data. Recommended practices include:
- Incremental updates rather than full-table refresh. The server sends changed fields, and the frontend performs partial rendering, reducing rendering overhead.
- Prefer server-side push. Push the latest snapshot to subscribed clients over a long-lived connection, avoiding high-frequency client polling.
- Data freshness indicators. The frontend should determine whether data is stale based on
updated_atand clearly display the update time in the interface, preventing users from misunderstanding old data.
7. Monitoring, Alerting, and Stability
Once a market data service is interrupted, the impact is direct and obvious, so observability is essential:
- Data freshness monitoring: Monitor the time since the last update for each instrument and alert when it exceeds a threshold. This is the fastest way to detect problems.
- Pipeline latency monitoring: Measure the time spent in ingestion, calculation, and storage separately to identify bottlenecks.
- Anomaly alerts: Trigger alerts when prices or percentage changes jump abnormally, making it easier for humans to intervene and verify.
- Dependency health checks: Continuously collect source connection status, cache hit rate, and storage write success rate.
On Baidu AI Cloud, this type of system can be built by combining general cloud capabilities such as log services, monitoring and alerting, message queues, object storage, and container platforms. Container platforms can host stateless ingestion and calculation services with elastic scaling, message queues can decouple layers, monitoring and alerting can cover freshness and latency metrics, and log services can preserve troubleshooting evidence. Product selection should be evaluated based on business scale and team capability. This section only discusses general architecture ideas.
8. Architecture Evolution and Security Compliance
After the system goes live, there are three main directions for evolution. First, scale out ingestion nodes horizontally by sharding instruments to reduce pressure on individual nodes. Second, introduce multi-level caching to further reduce read latency for hot instruments. Third, perform capacity planning and use load test data to guide resource allocation rather than scaling based on intuition.
Compliance considerations also require attention. Market data sources must have proper legal authorization, and redistribution or republication must comply with relevant agreements. For financial or asset-related data, avoid providing investment-advice-style interpretations; the system should be positioned as a data service rather than a decision-making tool. Data visible to users should retain source information and timestamps to ensure traceability and auditability.
9. Conclusion
The essence of building a market data service is turning a set of continuously changing numbers into trustworthy, timely, and scalable data capabilities. The key points include clear field modeling and precision handling, appropriate trade-offs between push and polling, layered and consistent storage design, a monitoring system centered on data freshness, and compliance awareness throughout the process. With this layered approach, it is possible to implement similar systems for digital assets, securities, or other tradable instruments using the same engineering skeleton, and to evolve smoothly as traffic grows.
