How to Rewrite a Core API With AI When You Have No Rust Experts: A Six-Step Method

Why a “do not touch” legacy service deserves a rewrite

Almost every medium-sized or larger engineering organization has a core service that is treated as untouchable by default. These services usually share a familiar pattern:

• The language choice is outdated. The original team chose a dynamic language for fast business validation. Years later, the service has become the entry point of the whole chain.
• Resource usage is already tight. CPU and memory usage stay high for long periods. Scaling is no longer “add capacity whenever needed”; it becomes “request capacity in advance.”
• Maintainers are shrinking. The engineers who understand the code move on, and onboarding cost for new maintainers is higher than the perceived cost of a rewrite.
• Change risk is asymmetric. The service carries all production traffic. One line can affect every downstream consumer.

The result is predictable: everyone agrees the service should be rewritten, but nobody wants to own the responsibility. The blocker is usually not technical difficulty alone. It is three hard-to-justify concerns:

1. No existing experts in the target language.
2. Behavior equivalence is hard to prove.
3. Performance gains are hard to quantify.

In a recent public technical interview, a platform engineering team addressed all three concerns at once. They rewrote a CPU-inefficient public inference invocation API — internally a monolithic service — using a systems programming language with ownership checks and borrowing rules. The team started with zero developers who had experience in that language. Most of the implementation was produced by AI coding tools, while humans handled system design and final validation.

After launch, the person responsible described the benefits as approximate, ballpark figures: CPU usage dropped by roughly 40% to 50%, first-token latency improved by roughly 10% to 20%, and longer-running workloads benefited more significantly.

First, three cold-water corrections

Before copying the process, calibrate three points that are easy to over-read from a headline:

1. The numbers are magnitude estimates, not a benchmark report. The interview used approximate language such as “ballpark.” This is a practitioner’s empirical estimate, not a reproducible benchmark with published test configuration or load-test details. Treat the numbers as an expected range, not a committed SLA.
2. “Zero experts” describes the starting point, not the whole journey. The interview also notes that engineers with relevant language experience joined during the process, and team members eventually became capable of maintaining the new system independently. The real path is: start with zero experts, then quickly build internal expertise.
3. Specific code-size figures circulating online do not have a verifiable primary source. This article does not rely on any unverified scale metrics.

The real conclusion

The most important lesson is not “AI can write systems-language code.”

The more important lesson is this:

When the implementer changes from a human to an AI, the project’s critical path shifts from “writing code” to “defining correctness and proving equivalence.”

Code generation has become a relatively cheap step. The expensive, decisive steps are:

• Defining the behavior contract.
• Building a differential validation system.
• Creating a parallel-evolution tracking mechanism.
• Designing a gray-release and rollback plan.

The six-step method below turns that critical path into an engineering process.

The six-step method at a glance

Step	Goal	Primary owner
1. Freeze the boundary	Define the project as “port only, no new features”	Human architect
2. Define behavior contracts	Turn “should behave the same” into executable assertions	Humans + AI
3. Design first	Humans make distributed-system design decisions	Humans
4. Implement and differentially validate	AI writes the implementation; shadow traffic proves equivalence	Mostly AI
5. Track feature alignment	Static analysis ensures newly added legacy features are implemented	Humans + tooling
6. Canary rollout	Move from small traffic to full traffic with rollback readiness	Humans

Step 1: Freeze the boundary and define the project as a pure port

One of the most overlooked but critical decisions in the interview is that the team deliberately restricted the project to a pure port and did not introduce new features.

This step is not only about “reducing risk.” It solves an attribution problem:

• If you introduce a new language, a new framework, and new features at the same time, any production anomaly becomes hard to attribute. Is it a porting defect, or a defect in the new functionality?
• Differential testing only works when both implementations are supposed to behave identically. If you add new features, the diff results always contain “expected differences,” and the validation system loses its power.

Executable boundary checklist

Put these rules into the project charter:

• Input/output protocols, field semantics, and the set of error codes remain unchanged.
• Thresholds for timeouts, retries, rate limits, and circuit breaking remain unchanged. If they need to change, open a separate project.
• No new parameters, endpoints, or default values.
• The categories of dependencies — storage, cache, message queues — remain unchanged. Only the client implementation may be replaced.

Any suggestion such as “optimize this while we are at it” goes into a Phase 2 backlog. This discipline matters more than the language choice.

Step 2: Turn “should behave the same” into executable assertions

This is the most human-intensive part of the methodology, and the part where AI is least helpful. The contract must be defined by the requirements owner, not inferred by the implementation.

Three types of contracts

Structural contracts describe fields, types, nullability, and enum values. They should be represented in a schema and validated in both directions.

{
  "output": {
    "content": [{"type": "text", "text": "string"}],
    "stop_reason": "enum(end_turn|max_tokens|tool_use|stop_sequence)",
    "usage": {"input_tokens": "int>=0", "output_tokens": "int>=0"}
  },
  "errors": [
    {"type": "invalid_request_error", "http_status": 400},
    {"type": "rate_limit_error", "http_status": 429},
    {"type": "overloaded_error", "http_status": 529}
  ]
}

Semantic contracts define what must be identical and what may differ. This is often missed, but it directly determines whether differential testing is usable.

Category	Judgment method
Structured fields such as stop_reason, usage, tool-call names, and arguments	Must be exactly equivalent
Text content	Sampling differences are allowed; exact character-by-character comparison is not valid when temperature > 0
Timestamps, trace_id, and instance names	Normalize and exclude
Floating-point values	Define relative or absolute tolerance
Unordered collection responses	Sort before comparison

Behavioral contracts describe state machines and boundary conditions. Examples include:

• If a streaming connection is interrupted, are remaining quota or credits rolled back?
• When max_tokens is exactly equal to the model limit, which branch is executed?
• If tool-call arguments are invalid, at which layer does the error occur?

These must be fixed with tests. Do not leave them to implementation freedom.

Record real production traffic as a corpus

Sample requests from production, desensitize them, and build a layered corpus:

• Single-turn short requests and long-context requests.
• Streaming SSE responses and non-streaming responses.
• Requests that trigger tool calls or function calls.
• Error paths: unauthorized, over quota, overly long input, invalid enums, and so on.
• Stress-test workloads: very long inputs, very long outputs, and concurrency spikes.

This corpus becomes the regression baseline for all later validation. Its coverage defines the upper limit of the correctness you can prove.

Step 3: Humans design, AI implements

The division of labor in the interview is clear: AI writes code, while humans make system-design decisions — because this is still fundamentally a distributed-system architecture problem.

This is the real difficulty of the migration. Moving from a monolithic dynamic-language service to a compiled systems-language service is not merely a syntax change. It changes the runtime model. Humans must make decisions such as:

1. Concurrency model. Single-threaded event loop with an async runtime, or multithreaded worker pool with message passing? This affects the ceiling of CPU savings and tail latency.
2. Memory ownership boundaries. Who owns the request context? When are response buffers released? How can whole-response copying be avoided in streaming scenarios? If streaming is implemented as “buffer everything, then send,” the latency benefit disappears.
3. Backpressure and queueing strategy. Should the entry point block, drop, or queue requests? What is the queue limit? How do timeouts propagate upward?
4. Observability instrumentation. Which metrics must align with the new architecture so that performance gains are measurable?
5. Failure modes and degradation paths. When a dependency times out, which error code should be returned? It must strictly match the legacy implementation.

The task description given to AI should be at this level of specificity, not merely “rewrite this Python service in Rust.”

Task: implement the request routing layer
Constraints:
- Use one async task per connection; do not block threads with shared mutable state.
- If request-body parsing fails, return 400 invalid_request_error; the error body must match the legacy implementation field by field.
- Streaming responses must be written out immediately when the first chunk is available; buffering the entire response is forbidden.
- Map all internal errors to 500; do not leak internal details in the response body.
- Expose per-request metrics: parse duration, queue-wait duration, first-chunk write duration, and total duration.
- Unit tests must cover: empty request body, oversized headers, invalid JSON, and mid-stream disconnection.

Drive AI implementation convergence with tests

The most effective pattern in practice is: write tests first, then let AI iterate until everything passes.

# 1) Generate golden samples from the legacy implementation
record_baseline --impl=legacy --corpus=./corpus --out=./golden
# 2) Let the AI implement and self-test
#    In CI, run the new implementation against the same corpus and diff it against golden
diff_check --impl=new --golden=./golden \
           --schema=contract.json \
           --tolerance=tolerance.yaml

A differential pass rate — for example, “997 of 1,000 samples are judged equivalent” — is a much stronger quantitative feedback signal for AI than vague instructions such as “optimize it further.”

Step 4: Use shadow traffic as core evidence of equivalence

One of the most useful lessons from the interview is this: build a complete new implementation, mirror a copy of production traffic into it, compare outputs from both sides, and only then gradually shift real traffic.

Shadow deployment topology

                         +--------------+
Real traffic ----> Edge gateway ----+ Legacy (primary) ----> Real response to users
                         +------+-------+
                                | Mirror (async / response discarded)
                         +------v-------+
                         + New (shadow) ----> Comparator ----> Difference sample store

Three hard requirements:

1. Shadow responses must never be returned to users, and they must not cause real side effects such as database writes, quota deductions, or message sends. This requires a “shadow mode” switch so external dependencies use read-only paths or local stubs.
2. Mirroring must be asynchronous and drop-tolerant. If the shadow path becomes overloaded, it should automatically downsample. It must not create backpressure on the primary path.
3. Comparison must cover streaming scenarios. Do not compare only the final concatenated output. You must also compare the semantics of the chunk sequence: the number of chunks may differ, but the concatenated content, stop reason, and usage statistics must be equivalent.

Tolerance and difference severity

Do not pursue “zero differences.” On an inference service with sampling randomness, that is unrealistic. The correct approach is severity-based classification:

• P0 blocking differences: different error codes, missing protocol fields, inconsistent usage statistics, inconsistent tool-call parameters, or panic/crash in the shadow path.
• P1 semantic differences: different stop reason for the same input, content length outside tolerance, or broken chunk semantics.
• P2 acceptable differences: text differences caused by sampling randomness, different timestamps or internal identifiers, and different performance numbers.

Process rules:

• P0 differences must be eliminated before moving to the next step.
• P1 differences require manual attribution: are they implementation deviations or sampling variation?
• P2 differences only require sampling monitoring.

tolerance:
  usage.input_tokens: {mode: exact}
  usage.output_tokens: {mode: relative, max_delta: 0.05}
  stop_reason: {mode: exact}
  content_text: {mode: semantic, min_similarity: 0.98}
  latency_ms: {mode: informational}
ignore_fields: ["created_at", "request_id", "server_instance"]

Step 5: Track feature alignment while the legacy service continues evolving

Migrations rarely happen in a vacuum. The legacy implementation keeps adding features. If the new implementation does not keep up, the cutover may silently lose capabilities.

Another practical detail from the interview is that the team used methods such as static analysis to confirm that newly added features in the legacy version had corresponding implementations in the new system.

Capability matrix as the single source of truth

Maintain a machine-readable capability matrix, generated automatically from code rather than relying on human memory.

# capability_matrix.yaml - generated by scanning the legacy implementation
features:
  - name: prompt_caching
    source_symbols: [legacy/cache.py::CacheLayer]
    status_in_new: implemented   # missing | implemented | intentionally_dropped
    tests_required: [test_cache_hit, test_cache_evict]
  - name: batch_endpoint
    source_symbols: [legacy/batch.py::BatchHandler]
    status_in_new: missing       # CI blocks the merge

How to implement static analysis

• Symbol-level alignment checks. Scan public endpoints, request parameters, and error-type enums in the legacy implementation. Check whether corresponding declarations exist in the new implementation. Warn on missing items.
• API change subscription. Add a hook to the legacy code-review workflow. If a publicly visible parameter or error code is added or modified, automatically create an “alignment task” in the migration project.
• Bidirectional contract tests. Use the same schema to validate both implementations. Any drift on either side will be exposed in CI.

A simplified parity-check script example:

def check_parity(legacy_ast, new_ast):
    missing = []
    for endpoint, params, errors in extract_public_surface(legacy_ast):
        new_surface = extract_public_surface(new_ast)
        if endpoint not in new_surface:
            missing.append(f"missing endpoint: {endpoint}")
            continue
        for p in params:
            if p not in new_surface[endpoint].params:
                missing.append(f"missing parameter: {endpoint}.{p}")
        for e in errors:
            if e not in new_surface[endpoint].errors:
                missing.append(f"missing error code: {endpoint} -> {e}")
    return missing   # non-empty result fails CI

Note: Automated checks can only cover interface surfaces that are statically recognizable. Business-logic branches still require human review. Do not treat static analysis as a complete proof.

Step 6: Canary rollout and rollback design

Shadow traffic proves that behavior is consistent under the same inputs. Canary rollout validates that the system remains stable under real traffic distribution and real load. The two cannot replace each other.

Rollout cadence

Stage	Traffic share	Observation focus	Pass condition
Shadow	0% (read-only mirror)	Difference rate, crash rate	P0 differences are zero, stable for 24–72 hours
Internal	<1% (allowlist)	End-to-end success rate, real user perception	No new error types
Canary A	1%–5%	Error rate, tail latency, resource usage	No significant degradation versus the legacy baseline
Canary B	20%–50%	Capacity headroom, dependency pressure, saturation	Sufficient scaling margin, no cascading effects
Full	100%	Long-tail tasks, cost curve	Legacy implementation remains available for rollback for N days

Four prerequisites for rollback

1. Switch granularity must be fine enough. Rollout should be controllable by endpoint, tenant, or request characteristics such as input length. Avoid all-or-nothing switches.
2. Keep the legacy implementation warm. Even after full rollout, retain legacy processes and images so rollback can happen in minutes.
3. State must be stateless or migratable. Session state, cache state, and deduplication records must either be shared or have a clear cutover strategy. Otherwise, rollback may introduce data inconsistency.
4. Rehearse the rollback. Practice a real rollback before launch. Do not rely on “it should work in theory.”

How to measure performance gains credibly

The performance numbers in the interview are attractive, but you should not copy them blindly. You must establish your own measurement framework. Otherwise, it is easy to be fooled by the feeling that things “look faster.”

Metrics must be attributable

• Resource efficiency: CPU time per request, maximum QPS per instance, peak memory per request, and GC pause ratio. Total CPU usage can be distorted by traffic variation.
• Latency distribution: P50, P95, P99, and P99.9. Measure time to first token and total duration separately. Averages have little decision value.
• Workload layers: short requests, long-context requests, and long-output requests. Systems languages often provide larger benefits for long-running tasks and streaming scenarios than for very short requests. This explains why the interview observed larger improvements on longer workloads.
• Cost accounting: cost per million requests on comparable instance types, not vague claims such as “we saved half the servers.”

Three measurement pitfalls

1. Prefer simultaneous comparison over before-and-after comparison. Run both implementations on the same set of requests in the same time window using shadow traffic. This is more reliable than comparing “last week” with “this week.”
2. Perform baseline warm-up and steady-state sampling. JIT, connection-pool warm-up, and cache cold/hot states can distort the first few minutes of measurements.
3. Record build and runtime configuration. Optimization levels, LTO settings, and async runtime worker-thread counts can materially affect results. Include them in the report, or the results will not be reproducible.

Common failure modes

Failure mode	Symptom	Countermeasure
Porting and adding features at the same time	Differences cannot be attributed; validation loses meaning	Clearly define a Phase 2 backlog in the project charter
Streaming implementation is buffered	First-token latency becomes worse	Make “first-chunk write duration” a top-level metric
Error paths are missing from the corpus	Main paths pass, but online error codes drift	Require the corpus to include every error branch
Only final outputs are compared	Broken chunk semantics go undetected	Compare chunk-sequence semantics, not only concatenated output
Shadow path affects primary path	Mirroring creates backpressure and production jitter	Use async mirroring, drop tolerance, and automatic downsampling
Parallel evolution becomes disconnected	Capabilities are silently lost after cutover	Use a capability matrix, CI blocking, and review hooks
No rollback rehearsal	Teams are afraid to switch back during incidents	Rehearse rollback before launch
Team capability is not accumulated	Nobody dares to modify the system after project completion	Deliberately build internal experts and run code walkthroughs

When not to use this approach

This method has boundaries. Defining them is more valuable than exaggerating its applicability.

• The algorithm or business logic is still being explored. If requirements are still changing, the “pure port only” assumption does not hold. Prototype first.
• Correctness cannot be automatically judged. If output quality can only be evaluated subjectively by humans and there is no stable baseline, differential validation cannot be built. Risk is uncontrollable.
• The bottleneck is not in the language runtime. If profiling shows that most time is spent in downstream model inference or network I/O, rewriting the language layer may deliver far less benefit than expected. Profile first, then decide whether to rewrite.
• The team cannot take ownership. AI can write the implementation, but it cannot absorb production incidents for you. At minimum, you need one person capable of making architectural decisions and a team willing to build the required language skills during the project.

Implementation checklist

Decision phase

• Complete performance profiling and confirm that the bottleneck is really in runtime or resource efficiency, not in downstream dependencies.
• Define the “pure port” boundary and establish a Phase 2 backlog.
• Assign an architecture owner and confirm the team is willing to invest in capability building.

Preparation phase

• Record and desensitize production traffic, and build a layered corpus covering all error branches.
• Write schema contracts, semantic tolerance configuration, and the ignore-field list.
• Build the differential comparator and difference-severity classification (P0/P1/P2).
• Design the shadow path: asynchronous mirroring, shadow-mode switch, and overload downsampling.

Implementation phase

• Humans complete design decisions for concurrency model, memory boundaries, and backpressure strategy.
• Let AI iterate in a test-driven way, using differential pass rate as the convergence signal.
• Introduce the capability matrix and static-analysis checks, and connect them to CI blocking.
• Bring P0 differences to zero and complete manual attribution for P1 differences.

Cutover phase

• Keep shadow traffic running stably for 24–72 hours.
• Progress through 1%, 5%, 50%, and 100% rollout, with quantitative pass conditions at each level.
• Keep the legacy implementation warm, and prepare rollback switches and rehearsals.
• Build a benefit report template that tracks workload layers, latency percentiles, and cost per request.
• Conduct a post-project review, recording the number of internal language experts and the code-walkthrough mechanism.

Conclusion

The real dividing line of this process is not whether AI generated the code. It is whether the team is willing to spend budget on proof of correctness: contract definition, corpus construction, differential comparison, capability tracking, and canary rollback.

These activities are expensive in traditional rewrites too. In the past, their cost was hidden by the assumption that “someone will write the code anyway.”

When implementation cost drops rapidly, validation cost becomes the new bottleneck. Teams that engineer their validation system will take on significantly lower rewrite risk than teams that rely on a few individual experts. On the other hand, if AI is treated merely as “free labor” and quantitative equivalence validation is skipped, the result is often a faster new system that nobody trusts.

As for the figures such as “CPU savings of about half” and “first-token latency improved by 10% to 20%,” treat them as a magnitude reference. The principle worth keeping is this:

Define what “correct” means first, let AI implement it, and then prove with data that it is correct.