Refactoring a Single-Point Centralized Architecture: From Hot Switching to High Availability

The Short Version

The risks in a single-point centralized architecture are concentrated in three areas: primary node switchover that is too rushed, unbalanced resource allocation, and a one-size-fits-all circuit-breaking strategy. When these problems combine, a single traffic peak can make the entire system unavailable. This article explains each issue, how to identify it, and how to refactor it.

1. Primary Node Switchover: Risks of an Unwarmed New Node

Primary node switchover is one of the highest-risk operations in a single-point architecture. If a new node takes over all traffic without sufficient validation, defects in exception handling, connection pool management, and cache warm-up will be exposed directly in production.

Common problems include:

• The new node lacks fallback logic for timeouts and retries, so threads are quickly exhausted when a downstream service becomes unstable.
• The local cache is empty, so the database receives a sudden load after switchover.
• Health checks only verify whether the process is alive, not whether dependent services are available.

A recommended approach is canary switchover. First send a small amount of traffic, such as 1% to 5%, to the new node. Observe error rate, latency, and resource utilization. Once stability is confirmed, gradually increase traffic. The following Kubernetes Deployment manifest is illustrative and shows rolling updates and readiness probe configuration.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: system-core
  namespace: prod
spec:
  replicas: 3
  selector:
    matchLabels:
      app: system-core
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0
      maxSurge: 1
  template:
    metadata:
      labels:
        app: system-core
    spec:
      containers:
        - name: system-core
          image: registry.example.com/system-core:1.4.2
          ports:
            - containerPort: 8080
          readinessProbe:
            httpGet:
              path: /health/ready
              port: 8080
            initialDelaySeconds: 30
            periodSeconds: 10
            failureThreshold: 3

This manifest is only an example. Actual parameters need to be adjusted according to the Kubernetes version, application startup time, and image contents. The readiness probe should check whether the database, cache, and downstream interfaces are available, not just whether the process exists.

During switchover, keep old-version replicas until the new version has passed the observation period. If problems occur, roll back to the previous version with kubectl rollout undo deployment/system-core -n prod.

2. Resource Allocation: Protection Components Competing with Business Workloads

Security auditing, log collection, and monitoring agents are necessary, but their resource usage needs limits. If these components share the same resource pool as business containers and no quotas are set, business containers will run short of resources during peak periods.

To determine whether allocation is unbalanced, measure the CPU and memory share of each type of component during the same period. These numbers should come from actual monitoring data, not experience-based estimates. If a daemon component consumes a large amount of resources during stress testing, optimize its sampling frequency and reporting method before considering limits.

Use ResourceQuota to limit the total resources of a namespace. The values below are examples only; actual quotas need to be calculated from stress-test results.

apiVersion: v1
kind: ResourceQuota
metadata:
  name: compute-resources
  namespace: prod
spec:
  hard:
    requests.cpu: "20"
    requests.memory: 40Gi
    limits.cpu: "40"
    limits.memory: 80Gi

A common sizing method is to take the actual business usage at the target peak, multiply it by a redundancy factor of 1.2 to 1.5, and then add the budget for protection components. At the same time, use LimitRange to set default requests and limits for individual containers, preventing any single container from consuming resources without bound.

3. Pseudo-Distributed Lock: Leader Election Failure

A pseudo-distributed lock here means that multiple nodes are deployed on the surface, but only one node can actually hold the primary identity. There are three common forms:

1. The primary node is hard-coded in configuration, with no automatic election mechanism.
2. The lease never expires, so other nodes cannot take over after the primary node fails.
3. There is no shared consensus store between nodes, and each node believes it is the primary.

Detection methods include:

• Check the renewal time of the leader-election resource, such as a Kubernetes Lease object or a key in etcd. If it has not been updated for a long time, the election mechanism has failed.
• In a test environment, isolate the primary node’s network and observe whether other nodes can elect a new primary after the lease times out.
• Record the time taken by one switchover and compare it with the design target.

For refactoring, use a proven consensus approach, such as Raft, on which both etcd and Consul are based, or a variant of Paxos. The lease timeout needs to be determined together with network jitter data. Too short a timeout causes frequent switchovers; too long a timeout extends unavailability.

4. Circuit Breaking and Rate Limiting: From Blanket Blocking to Tiered Admission

When system pressure exceeds a threshold, a common approach is to reject all new requests. This harms normal traffic and can leave the system at low load for a long time after recovery.

A more reasonable approach is layered handling:

• Rate limiting: use token bucket or leaky bucket algorithms to control the ingress rate and smooth out traffic bursts.
• Circuit breaking: when the error rate of a downstream service exceeds a threshold, open the circuit breaker so requests fail fast, and periodically probe whether the downstream service has recovered.
• Degradation: pause non-core functions such as recommendations and statistical reports to keep core paths available.
• Priority: reserve quota for critical business so core requests pass first.

Circuit-breaking parameters need to be set based on business data, including the error-rate threshold, statistics window length, and probe ratio in the half-open state. Default values usually cannot be applied directly; they should be tuned gradually through stress testing and drills.

5. Refactoring Path

For the issues above, refactoring can proceed in three steps:

1. Break single-point dependencies: deploy across multiple availability zones and make leader election and switchover automatic, without manual intervention.
2. Out-of-band protection: move heavy components such as auditing, encryption, and logging out of the main business request path, using sidecars or bypass collection to reduce request latency.
3. Flexible circuit breaking: replace global blocking with rate limiting, circuit breakers, and degradation.

Each step should have verifiable completion criteria. The following checklist can be used as a review basis:

Check item	Pass criteria	Verification method
Single-point identification	List all core components that have only one instance	Architecture diagram and dependency review
Capacity estimation	Peak usage is backed by stress-test data	Stress-test report
Switchover drill	Automatic switchover after primary failure within the target time	Regular fault injection
Circuit-breaking parameters	Thresholds, windows, and probe ratios are based on historical metrics	Compare with historical monitoring data
Rollback validation	Return to the previous stable version within the agreed time	Rollback drill records

6. Risk Notes

• Canary releases extend the release cycle, so there is a tradeoff between efficiency and risk.
• A multi-active architecture must handle data consistency. Cross-zone write latency and conflict resolution strategies should be determined during the design phase.
• Cross-zone deployment increases bandwidth and operational costs, which need to be estimated separately.
• Fault drills should be scheduled during off-peak hours, with relevant teams notified in advance.
• Every refactoring step needs a rollback plan, and its feasibility should be verified in drills.

Summary

The problem with a single-point architecture is usually not a defect in one component, but an imbalance in three design areas: switchover, resource allocation, and protection. Refactoring can proceed in this order: first identify single points, then complete leader election and canary switchover, then set resource quotas, and finally adjust the circuit-breaking strategy. Validate each step with monitoring data so that the system degrades gracefully during failures instead of collapsing entirely.