How to Build an Asynchronous Event-Driven Architecture: State Management and Resource Scheduling

This article is intended for backend developers and software architects. It focuses on two common challenges in asynchronous event-driven architecture (EDA): how to move business state through the system reliably, and how to isolate and schedule resources under high concurrency. The parameter values in this article are examples. In production, they should be adjusted according to load-test results and business service-level objectives (SLOs).

1. Resource Isolation and Graceful Degradation: Define the SLO First

System design should start with a service-level objective. For a core order-placement flow, for example, you might define 99.95% availability and a P99 latency target of no more than 300 ms. All isolation and degradation strategies should support these objectives.

Three common isolation techniques are widely used:

1. Thread-pool isolation: Assign each downstream dependency its own thread pool and bounded queue. If a dependency becomes slow, only its thread pool is exhausted, while calls to other dependencies remain unaffected. When the queue is full, reject requests immediately to prevent unlimited buildup.

2. Semaphore isolation: Limit the number of calls that can be in progress at the same time without creating additional threads. This approach is suitable for short-lived calls to fast internal services and has lower overhead than thread-pool isolation.

3. Circuit breaking and fallback: Measure the error rate and slow-call ratio over a sliding window. When a threshold is exceeded, return a fallback result directly instead of calling the downstream dependency.

An example configuration for a recommendation service might use a thread pool with core=20, max=40, and a queue length of 100, with a 200 ms timeout for each call. The circuit breaker could use the following rules:

• In a 20-second sliding window, at least 50 requests must have been received.
• The slow-call ratio, where a slow call exceeds 500 ms, is greater than 50%; or
• The error rate is greater than 30%.

When triggered, the circuit remains open for 30 seconds. It then enters a half-open state and allows five probe requests through. The circuit is restored only if all five probes succeed.

Before enabling a fallback or degradation switch, verify three things:

1. Monitoring coverage: Monitoring must include the fallback hit rate, dependency error rate, active thread count, and queue depth.
2. Defined rollback conditions: If the core success rate does not recover to above 99.9% within five minutes after degradation begins, or if the degradation ratio exceeds 20% for 10 consecutive minutes, escalate the issue for manual handling.
3. Protection against lost writes: Write operations must not be lost during degradation. A common approach is to place write requests in a compensation queue and replay them after the dependency recovers. Replay must be idempotent.

2. Event Modeling: Fields, Idempotency, and Ordering

Event sourcing records every state change as an immutable event instead of updating only the final state in a database. This supports auditing, replay, and state reconstruction.

An event table should include fields such as:

• event_id: A globally unique identifier, generated with UUIDv7, a Snowflake-style algorithm, or an equivalent mechanism.
• aggregate_id and aggregate_type: Identify the business object to which the event belongs.
• version: A monotonically increasing version within the aggregate, used for optimistic concurrency control.
• event_type: For example, OrderPaid or OrderShipped.
• occurred_at: The time when the business event occurred. Keep it separate from the database insertion time.
• payload: The event data, preferably stored as JSON with an accompanying schema_version.
• trace_id: Used for troubleshooting across services.

Idempotency is a basic requirement for event consumers. Message systems commonly provide at-least-once delivery, so the same event may be consumed more than once.

A recommended approach is to deduplicate events by event_id and create a unique index on (aggregate_id, version) in the business table. The deduplication record and the business result should be committed in the same database transaction. Otherwise, the business operation might succeed while the deduplication record is lost, allowing the event to be processed again.

Ordering is meaningful only within an individual aggregate. In practice, use the same aggregate_id as the partition key—for example, as the Kafka key—so that events for the same object are routed to the same partition and remain ordered within that partition.

Global ordering across different partitions is not guaranteed. Business logic should not depend on the relative order of events belonging to different objects.

If a consumer detects a gap in the version sequence, it should temporarily store the later event and wait for the missing version to arrive. If the wait exceeds a defined limit, such as 30 seconds, the system should raise an alert and route the case for manual investigation.

Snapshots help control replay costs. A snapshot can be generated after every 100 events.