Designing and Implementing a Multi-Backend Gateway for Local LLMs
As open-source LLMs mature and free or low-cost APIs become available, running inference locally has become a default choice for many teams and individual developers. The friction starts when different inference engines expose different HTTP interfaces: inconsistent endpoint paths, different default ports, mismatched request and response shapes, and inconsistent sampling parameter names and ranges. This article walks through a practical design for a local LLM gateway that absorbs those differences in a single adapter layer, including a configuration-driven backend registry, correct Server-Sent Events (SSE) streaming with buffered chunk parsing (the usual cause of "infinite loop" hangs), unified system prompt and parameter handling, safe rendering of lightweight markup, responsive UX details, a troubleshooting checklist, and security recommendations for moving a local prototype toward an operable production system.
1. Background: Local LLMs and the Fragmentation Across Inference Backends
Over the past couple of years, open-source LLMs have improved rapidly, and the surrounding free or low-cost APIs have made “deploy locally + call locally” a first choice for many teams and individual developers. In practice, though, developers quickly hit a concrete problem: there is no shared standard for how local inference engines expose an HTTP interface.
Common differences include:
- Inconsistent endpoint paths. For the same “chat completion” capability, one backend may use
/v1/chat/completions, another/api/chat, and another/completion. - Different default ports. Different tools listen on ports such as 11434 or 1234 by default, which is easy to confuse when switching between them.
- Different request and response structures. Field names, the shape of the message array, and the delimiters used in streaming chunks all vary.
- Different parameter names and ranges. Sampling parameters such as temperature and top-p have different valid ranges and defaults on different backends.
Without an abstraction, every new backend means editing business code again, and maintenance cost grows linearly with the number of backends. The core question this article addresses is: how do you collapse all of these differences behind a single unified gateway layer?
2. Core Design Goals and a Layered Architecture
A usable local inference gateway should state a few design goals explicitly:
- Pluggable backends. Adding a backend should only mean adding an adapter configuration, with no changes to the layers above it.
- Unified interface. Regardless of which backend runs underneath, callers always see the same request and response structure.
- Streaming-friendly. Token-by-token output for a typewriter effect, which noticeably improves interaction during long generations.
- Adaptive parameters. Parameters such as temperature and top-p are automatically mapped and validated against the active backend.
- Complete experience. System prompts, loading states, error messages, and a responsive layout.
The corresponding layering can be abstracted as:
┌──────────────────────────────────────┐│ Interaction layer (UI/callers) │├──────────────────────────────────────┤│ Unified interface layer (normalize) │├──────────────────────────────────────┤│ Backend adapter layer (paths/ports/fields)│├──────────────────────────────────────┤│ Actual inference backend(s), swappable │└──────────────────────────────────────┘
The key point: upper layers only know the “unified interface” and are completely unaware of backend differences. Every difference is confined to the backend adapter layer.
3. The Backend Adapter Layer: Configuration Instead of Hardcoding
The adapter layer is the heart of the gateway. A configuration table describing how each backend is reached is preferable to scattering if/else branches through the code.
const BACKENDS = {engineA: {name: "Backend A",baseURL: "http://localhost:11434",chatPath: "/api/chat",stream: true,// Parameter mapping: unified name -> the backend's actual fieldparamMap: {temperature: "temperature",topP: "top_p",},},engineB: {name: "Backend B",baseURL: "http://localhost:1234",chatPath: "/v1/chat/completions",stream: true,paramMap: {temperature: "temperature",topP: "top_p",},},};// Build a backend request body from unified parametersfunction buildPayload(cfg, messages, opts) {const body = { model: opts.model, stream: cfg.stream, messages };for (const [k, v] of Object.entries(opts.sampling || {})) {const target = cfg.paramMap[k];if (target) body[target] = v; // only inject fields this backend supports}return body;}
This design has two benefits: adding a backend requires only one more configuration block, and parameter mapping is managed in one place. If a backend does not support a given parameter, you simply leave it unmapped instead of triggering an error.
4. Handling Streaming Responses and Avoiding the “Infinite Loop” Trap
Streaming (token-by-token output) is essential to local inference UX. The underlying mechanism is straightforward: the backend pushes a series of chunks starting with data: in Server-Sent Events (SSE) style, and the frontend renders as it receives.
Most “direct infinite loop” problems in hands-on implementations trace back to one detail: chunk boundaries are not handled correctly. Because of network fragmentation, a single complete JSON payload may arrive split across several chunks, or one chunk may contain multiple records. If you call JSON.parse directly after every read(), it will throw repeatedly on partial data and spin in a loop.
The correct approach is to maintain a buffer and parse only when a complete delimiter (typically \n\n) is present:
async function streamChat(cfg, messages, opts, onToken) {const resp = await fetch(cfg.baseURL + cfg.chatPath, {method: "POST",headers: { "Content-Type": "application/json" },body: JSON.stringify(buildPayload(cfg, messages, opts)),});if (!resp.ok) throw new Error(`Backend returned an error status: ${resp.status}`);const reader = resp.body.getReader();const decoder = new TextDecoder();let buffer = "";while (true) {const { done, value } = await reader.read();if (done) break;buffer += decoder.decode(value, { stream: true });// Process only on a complete delimiter, avoiding repeated parsing of partial JSONconst parts = buffer.split("\n\n");buffer = parts.pop() || ""; // the last segment may be incomplete; keep it for the next roundfor (const part of parts) {const line = part.replace(/^data:\s*/, "").trim();if (!line || line === "[DONE]") continue;const json = JSON.parse(line); // the data is guaranteed complete hereconst token = json.choices?.[0]?.delta?.content ?? "";if (token) onToken(token); // append to the UI to create the typewriter effect}}}
This code handles three common mistakes correctly: streaming reads (getReader), buffered concatenation (buffer), and boundary detection (split("\n\n")). Once those three steps are in place, the infinite-loop problem is essentially eliminated.
5. Unifying System Prompts and Generation Parameters
A system prompt defines the model’s role, tone, and constraints, and is standard in every chat application. At the gateway layer, it should be injected as the first entry in the message array:
function withSystemPrompt(system, userMessages) {return [...(system ? [{ role: "system", content: system }] : []),...userMessages,];}
As for sampling parameters like temperature and top-p, they control randomness and diversity in generation: lower temperature produces more deterministic, conservative output, while higher values diverge more; top-p truncates candidate tokens by cumulative probability mass. The gateway should pass values through while validating them — map the user’s settings to the active backend via the adapter’s paramMap, and warn when a value falls outside a sensible range instead of letting the underlying backend fail.
6. Rendering Lightweight Markup Safely
LLM output is usually lightweight markup: symbols indicating headings, lists, code blocks, and so on. Rendering it in full via innerHTML carries two risks: the raw symbols may be shown as-is, and there is a potential script injection vector. A “simple parsing plus escaping” strategy is recommended:
- First HTML-escape the text, replacing characters such as
<,>, and&to eliminate injection. - Then convert markup symbols (
#for headings,```for code blocks,-for lists) into the corresponding tags using lightweight rules. - Handle code blocks separately, preserving indentation and monospaced font so they are not mistaken for ordinary paragraphs.
One thing to stress: escaping must happen before markup parsing. Reversing that order defeats the injection protection.
7. Modern Interaction and Experience Design
Engineering usability matters as much as the technical implementation. A mature gateway UI should include:
- Responsive layout. Configuration items (backend switching, system prompt, parameter sliders) in a sidebar, the conversation stream in the main area, stacking automatically on narrow screens.
- Loading and error states. Disable input during a request and show a “Generating” indicator; on failure, display the specific reason prominently (connection failure, out-of-range parameter, and so on).
- State isolation. Reset the session when switching backends so context from the previous backend does not contaminate the current conversation.
- Dark theme. Manage the palette through CSS variables so contrast and readability can be adjusted in one place.
8. Troubleshooting Checklist
When putting this into practice, work through the following items:
- Connection failure. Confirm the backend process is running and that the port matches the configuration.
- Blocked by CORS. When a browser talks directly to a local backend, allow the corresponding origin on the backend, or route requests through a local proxy.
- No streaming output or an infinite loop. Check whether chunks are split on the delimiter and whether the trailing incomplete fragment is retained.
- Parameters not taking effect. Confirm the adapter’s
paramMapcovers that parameter and that the backend supports it. - Garbled Chinese text. Decode the stream with
TextDecoderandstream: trueso multi-byte characters are joined correctly across chunks.
9. Best Practices and Security Recommendations
A few engineering recommendations to move a solution from “it runs” to “it’s good to use”:
- Keep the adapter layer clean. Backend differences belong only in configuration; business code should not be aware of any specific backend.
- Validate in both directions. Validate user-supplied parameters, and defend against unexpected structures in backend responses.
- Escape by default when rendering. Any model text entering the page is escaped first, then parsed.
- Local-first security boundary. A local gateway should listen only on the loopback address by default. If LAN access is required, explicitly assess access control and authentication risks so the inference service is not reachable by unauthorized parties.
- Evolve incrementally. Leave extension points for authentication, rate limiting, logging, and monitoring alerts, so the local prototype can later integrate general-purpose cloud capabilities such as object storage, message queues, log services, and container platforms, and grow smoothly into an operable production system.
In short, building a unified API gateway for local LLMs means using the adapter pattern to converge backend differences, buffered parsing to keep streaming stable, and a unified abstraction to carry prompts and parameters. With this design in hand, upper-layer applications stay stable and maintainable no matter how the underlying inference backend changes — a key step in moving localized LLM work from individual experiments to team engineering practice.
