Points Settlement: State Machine, Idempotency, and Concurrency in Practice

    Points, rebates, commissions, and coupon redemption are fragile. They can be over-issued, miscounted, or counted twice, and any design flaw is amplified under peak traffic. This article walks through how to build a points settlement engine that can't over-issue, can't under-calculate, can't double-count, and won't get stuck: model the workflow as a state machine with explicit terminal states, derive domain models and contract tests from that spec, decouple upstream services with event-driven processing and the Outbox pattern, and guard concurrency with idempotency keys, unique-index conflict handling, optimistic locking, retry backoff, and a dead letter queue. It also covers the common mistakes and the canary, reconciliation, and rollback practices required before rolling out any money-touching code.

    Why Points Settlement Is the Easiest Place to Lose Money

    Points, rebates, commissions, and coupon redemption are all about the precise movement of entitlements or funds. These systems tend to fail in four ways: they can’t over-issue, can’t miss a calculation, can’t count twice, and can’t get stuck. During peak scenarios such as big promotions, flash sales, bulk grants, and stacked campaigns, any design flaw can be amplified by concurrency and turn into a financial loss incident.

    So when you ask how to do points settlement, what the common mistakes are, or which idempotency approach to choose, checking whether a single API works is not enough. The design has to cover the state machine, event contracts, idempotency, transactions, compensation, reconciliation, canary releases, and rollback as a whole. AI coding tools can speed up code generation, test completion, and defect localization, but they still require clear specs, executable tests, strict human review, and a controlled canary validation process. Tools can’t replace risk controls for money, and they can’t replace reconciliation and rollback plans.

    Step 1: Break Down the Requirements and Model the State Machine

    The biggest mistake in building a settlement engine is jumping straight to database tables, APIs, and SQL. Once concurrency rises, without state constraints the data flow will quickly spiral out of control. The first step in the architecture is to draw the state machine and decide, for each state, whether it is a terminal state, whether it can be retried, and whether it can be handled manually.

    A settlement task can be modeled as a clearing_task. Its core fields include task ID, order ID, user ID, points amount, business type, idempotency key, status, version number, failure count, and next retry time.

    A suggested state machine:

    1. clearing_task_states:
    2. PENDING: # Awaiting settlement
    3. CALCULATING: # Calculation in progress
    4. SETTLED: # Settlement completed
    5. SETTLE_FAILED: # Settlement failed, eligible for compensation
    6. RETRYING: # Automatic retry in progress
    7. COMPENSATING: # Compensation in progress
    8. MANUAL_REVIEW: # Escalated for manual handling
    9. CLOSED: # Manually confirmed as failed and closed; cannot continue

    One point deserves special attention: SETTLE_FAILED should not be designed as an immutable terminal state. Settlement can fail because of timeouts, dependency outages, a temporarily unavailable ledger, or errors from a third-party rebate API, so every failure needs a path forward. A more sensible choice is to make SETTLED and CLOSED the only terminal states, while SETTLE_FAILED is an intermediate failure state that leads into the compensation flow.

    Transitions can be expressed as a machine-readable spec that serves as a single source of truth for code, tests, and documentation. For example:

    1. transitions:
    2. - { from: PENDING, to: CALCULATING, on: start_calculate }
    3. - { from: CALCULATING, to: SETTLED, on: calculate_success }
    4. - { from: CALCULATING, to: SETTLE_FAILED, on: calculate_error }
    5. - { from: SETTLE_FAILED, to: RETRYING, on: retry_allowed }
    6. - { from: RETRYING, to: CALCULATING, on: retry_start }
    7. - { from: SETTLE_FAILED, to: MANUAL_REVIEW, on: retry_exhausted }
    8. - { from: MANUAL_REVIEW, to: COMPENSATING, on: manual_compensate }
    9. - { from: MANUAL_REVIEW, to: CLOSED, on: manual_close }
    10. - { from: COMPENSATING, to: SETTLED, on: compensate_success }
    11. - { from: COMPENSATING, to: CLOSED, on: compensate_failed_and_confirmed }

    The benefit is that every illegal transition can be blocked in code. For example:

    • PENDING must not jump straight to SETTLED;
    • CLOSED must not go back to CALCULATING;
    • an ordinary event must not change SETTLED again.

    Step 2: Generate the Domain Model and API Contracts

    Once the state machine is in place, turn the spec into domain models, state enums, and transition guards. At this stage, the goal is code that is structured, verifiable, and introduces no invented behavior.

    Example code:

    1. from enum import Enum
    2. class TaskStatus(str, Enum):
    3. PENDING = "PENDING"
    4. CALCULATING = "CALCULATING"
    5. SETTLED = "SETTLED"
    6. SETTLE_FAILED = "SETTLE_FAILED"
    7. RETRYING = "RETRYING"
    8. COMPENSATING = "COMPENSATING"
    9. MANUAL_REVIEW = "MANUAL_REVIEW"
    10. CLOSED = "CLOSED"
    11. VALID_TRANSITIONS = {
    12. TaskStatus.PENDING: {TaskStatus.CALCULATING},
    13. TaskStatus.CALCULATING: {TaskStatus.SETTLED, TaskStatus.SETTLE_FAILED},
    14. TaskStatus.SETTLE_FAILED: {TaskStatus.RETRYING, TaskStatus.MANUAL_REVIEW},
    15. TaskStatus.RETRYING: {TaskStatus.CALCULATING},
    16. TaskStatus.MANUAL_REVIEW: {TaskStatus.COMPENSATING, TaskStatus.CLOSED},
    17. TaskStatus.COMPENSATING: {TaskStatus.SETTLED, TaskStatus.CLOSED},
    18. TaskStatus.SETTLED: set(),
    19. TaskStatus.CLOSED: set(),
    20. }
    21. def can_transition(src: TaskStatus, dst: TaskStatus) -> bool:
    22. return dst in VALID_TRANSITIONS.get(src, set())
    23. def apply_event(task, dst: TaskStatus):
    24. if not can_transition(task.status, dst):
    25. raise ValueError(f"Illegal state transition: {task.status} -> {dst}")
    26. task.status = dst

    Alongside this code, add contract tests that focus on three categories of cases:

    1. Valid transitions, such as PENDING -> CALCULATING -> SETTLED;
    2. Invalid transitions, such as PENDING -> SETTLED, SETTLED -> CALCULATING, and CLOSED -> RETRYING;
    3. Idempotency key validation: null values, overly long values, invalid characters, and duplicate keys.

    If the project uses code generation tools, pin dependency versions explicitly, such as Python 3.10, pytest, Pydantic, the database driver, and the message queue client. This avoids generated code that doesn’t match the runtime environment.

    Step 3: Event-Driven and Asynchronous Processing

    Under traffic spikes, a settlement engine can’t process everything synchronously and serially. A sound approach is event-driven processing with asynchronous decoupling: order events are first written to a message queue or a local event table, and the settlement service consumes them asynchronously.

    The typical flow is as follows:

    1. An order is created or paid successfully;
    2. In a local transaction, the order service writes the order record and a record to the local event table outbox_event;
    3. An asynchronous dispatcher publishes the event to the message queue;
    4. The settlement engine consumes the event and creates a clearing_task;
    5. Calculation is triggered and the task status is updated;
    6. After calculation succeeds, the points or ledger entries are written back;
    7. If any step fails, the task enters a failure, retry, compensation, or manual handling flow.

    Events should use an explicit schema so consumers don’t depend on the upstream database structure. For example:

    1. {
    2. "event_id": "evt_20250101_0001",
    3. "event_type": "ORDER_PAID_SUCCESS",
    4. "order_id": "O202501010001",
    5. "user_id": "U10001",
    6. "activity_id": "ACT_POINTS_2025",
    7. "points": 100,
    8. "idempotency_key": "sha256:O202501010001:ACT_POINTS_2025:PAID",
    9. "occurred_at": "2025-01-01T10:00:00Z",
    10. "source": "order-service",
    11. "trace_id": "trace_abc123"
    12. }

    Consumers must not assume each message arrives only once. Duplicate delivery, out-of-order delivery, and retries after failures are all the norm. That’s why you need a dead letter queue, a retry queue, failure alerts, and reconciliation jobs.

    Idempotent consumer pseudocode:

    1. def handle_order_paid_event(event):
    2. task_id = event["order_id"]
    3. idem_key = event["idempotency_key"]
    4. with db.transaction():
    5. idempotent = insert_idempotent_record(idem_key, event["event_id"])
    6. if not idempotent.inserted:
    7. return idempotent.existing_result
    8. task = get_or_create_clearing_task(task_id, event)
    9. if task.status == TaskStatus.CLOSED:
    10. return "closed"
    11. if task.status == TaskStatus.SETTLED:
    12. return "settled"
    13. task.status = TaskStatus.CALCULATING
    14. update_task_with_optimistic_lock(task)

    If the order service uses local transactions together with a message queue, consider the Outbox pattern: write the business data and the event data in the same local transaction, then have a separate dispatcher send the messages. This avoids the “database write succeeded but message publish failed” case that would cause the settlement to be missed.

    Step 4: Distributed Locks, Idempotency, and Resolving Concurrency Conflicts

    Async and distributed deployments mean the same task may be picked up at the same time by multiple consumers and multiple instances. Here you have to solve for idempotency, concurrency conflicts, and automatic recovery from failures.

    1. Idempotency key generation rules

    An idempotency key can’t rely on the order number alone. When campaigns are stacked, rewards are issued multiple times, settlement is split into installments, or different business types are involved, the order number may not be unique enough. It’s better to combine fields that carry business meaning:

    1. sha256(user_id + order_id + business_type + activity_id + event_version)

    For example:

    1. sha256("U10001:O202501010001:POINTS_REFUND:ACT_2025:V1")

    Once an idempotency key is generated, persist it with a unique index. Before processing, insert the idempotency record first; a successful insert means you’ve won the claim.

    2. Handling unique key conflicts

    It’s not enough to rely on duplicate keys throwing exceptions and assume the event was already processed. The right approach is to catch the unique key violation and turn the exception into a business result:

    1. INSERT INTO clearing_idempotent (idempotency_key, status, created_at)
    2. VALUES (?, 'PROCESSING', NOW());

    If the database returns a unique key conflict, check the existing record’s status:

    • Status SUCCESS: return success directly;
    • Status PROCESSING: the current instance doesn’t process it again. Retry later or return “processing”;
    • Status FAILED: decide whether to claim it again according to the failure policy;
    • Unexpected status: escalate to manual review or raise an alert.

    3. Transaction boundaries

    Keep transaction boundaries short and wrap only the state transition and the idempotency record. Don’t include long-running remote calls. For example, don’t call external rebate APIs, third-party payment APIs, or SMS APIs inside a transaction. If an external call fails, you can’t simply roll back the database transaction, because the external system may have already produced side effects that can’t be undone.

    A better approach is to let the local transaction complete the state change, then drive external calls to closure through compensation jobs, message retries, or manual handling.

    4. Optimistic locking and retry backoff

    Optimistic locking uses a version number to prevent one update from overwriting another:

    1. UPDATE clearing_task
    2. SET status = 'SETTLED',
    3. version = version + 1
    4. WHERE task_id = ?
    5. AND version = ?
    6. AND status = 'CALCULATING';

    If the affected row count is 0, the version has changed or the status has already transitioned. Treat this as a conflict and re-read the status.

    Conflict retries shouldn’t spin in a tight loop. Use exponential backoff and a maximum number of attempts:

    1. Attempt 1: retry immediately
    2. Attempt 2: wait 200ms
    3. Attempt 3: wait 800ms
    4. Attempt 4: wait 3s
    5. After 5 attempts: move to SETTLE_FAILED or the dead letter queue, and alert

    5. Distributed locks and alerting

    For resources that truly need to be serialized, you can use locks backed by Redis, ZooKeeper, or the database. However, you have to account for lock expiration, renewal, reentrancy, and accidentally deleting someone else’s lock. A more robust option is optimistic locking combined with an idempotency table, which keeps you from holding locks for long periods.

    You also need monitoring metrics:

    • Settlement success rate
    • Average processing time
    • P99 latency
    • Retry counts
    • Dead letter volume
    • Idempotency conflict rate
    • Optimistic lock conflict rate
    • Reconciliation discrepancy count

    When a metric crosses its threshold, trigger circuit breaking, rate limiting, or an alert.

    Summary: Methodology Matters More Than Tools

    The core path of a points settlement engine comes down to five steps:

    1. Build the state machine first, with clear states, transitions, and terminal states;
    2. Then generate the domain model and contracts, backed by tests for invalid transitions;
    3. Use event-driven processing and the Outbox pattern to decouple upstream business services from settlement;
    4. Use an idempotency table, unique keys, optimistic locking, retry backoff, and a dead letter queue to hold the line under concurrency;
    5. Finally, support it with canary releases, reconciliation, rollback, backups, and load tests in an isolated environment.

    Common mistakes include:

    • Missing idempotency
    • Locks with a granularity that’s too coarse
    • Events without enough context
    • Failure states mistakenly designed as terminal states
    • No Outbox
    • No reconciliation
    • No retry backoff
    • No manual handling entry point
    • Load tests that pollute production data

    A final word on financial and data risk: load-test any write involving points, entitlements, or commissions in staging or an isolated environment first. When you ship, roll out in a canary by user, campaign, or merchant, and prepare full reconciliation, discrepancy fixes, a rollback plan, and data backups. Automated fixes and AI coding tools can improve efficiency, but they can’t replace human review, risk approvals, and the safety boundaries around money.