Build a Multi-Model Coding Assistant with a Single-Board Computer

1. Why Add a Model Gateway to a Coding Assistant

Modern AI coding tools often support code completion, project analysis, file editing, and command assistance. However, their default model integration is usually relatively fixed. This creates several practical challenges for developers:

1. Limited model flexibility
   Different models vary in code generation, long-context understanding, multilingual communication, debugging, and response speed. A single model rarely performs equally well in every scenario.

2. Different API protocols
   Model services may use different endpoints, authentication methods, message structures, and response formats. If the client integrates with every service directly, maintenance costs continue to grow.

3. Local models are difficult to integrate consistently
   Models running on a local network may require additional port exposure, request transformation, and access control. Not every coding tool can identify or use them directly.

4. Testing and migration are inconvenient
   When developers replace a model provider or move a deployment, a tightly coupled client and backend require extensive configuration changes.

A general solution is to add a model gateway between the client and model services. The client connects only to the gateway. The gateway routes requests to the configured model and converts responses into a format the client can understand.

The architecture can be represented as follows:

AI Coding Tool
      │
      │  Unified API
      ▼
Local Model Gateway
      │
      ├── Cloud Model Services
      ├── LAN Model Services
      └── Local Inference Services

A low-cost single-board computer is not intended to perform complex model inference in this setup. Instead, it acts as an always-on, low-power, and easy-to-maintain relay node. This reduces deployment costs while centralizing model configuration, access control, and logging.

2. Prepare the Runtime Environment

This guide uses an ARM-based single-board computer running Linux as an example. Other mini PCs, virtual machines, or home servers with Python support can use a similar approach.

Recommended prerequisites include:

• A Linux device with a stable network connection;
• Python 3.9 or later;
• Available model service endpoints;
• The corresponding access keys;
• A regular user account for running the gateway;
• A client that can access the device over the local network.

First, update the system packages:

sudo apt update
sudo apt upgrade -y

Check the Python version:

python3 --version

If the virtual environment module is not installed, add it with:

sudo apt install -y python3-venv python3-pip

For production deployments, avoid installing third-party dependencies directly into the system Python environment. Mixing system components with project dependencies can cause version conflicts and makes future removal or migration more difficult.

3. Create an Isolated Project Directory

Create a dedicated directory for the model gateway:

mkdir -p ~/model-gateway
cd ~/model-gateway

Create a virtual environment:

python3 -m venv .venv

Activate it:

source .venv/bin/activate

After activation, the environment name typically appears in the shell prompt. Python and pip commands will then use the versions installed in the project environment first.

You can verify the paths with:

which python
which pip

If the output points to ~/model-gateway/.venv/, the virtual environment is active.

To exit the environment:

deactivate

To enter it again later:

cd ~/model-gateway
source .venv/bin/activate

4. Install a Compatible API Gateway

The model gateway is responsible for:

• Receiving client requests;
• Identifying the requested model;
• Converting the unified request format into the format required by the target service;
• Adding authentication information;
• Forwarding requests and returning responses;
• Recording access logs and errors when necessary.

Install the gateway dependencies using the virtual environment’s pip:

python -m pip install --upgrade pip
python -m pip install "compatible-api-gateway[proxy]"

Because package names differ between implementations, use the official documentation for the gateway you select. The placeholder name “compatible API gateway” is used here to explain the architecture without binding the guide to a specific tool.

After installation, check the dependencies:

python -m pip list

On a device with limited memory, disable unnecessary background services and avoid running model inference, databases, and multiple gateway processes at the same time.

5. Configure Model Routing

Create a configuration file in the project directory:

nano gateway-config.yaml

A possible structure is:

model_list:
  - model_name: coding-default
    provider:
      model: your-model-name
      api_base: https://your-model-service.example/v1
      api_key: ${MODEL_API_KEY}

The key fields are:

• model_name: The logical name used by the client. It can be customized;
• provider.model: The model name recognized by the target service;
• api_base: The API endpoint of the target model service;
• api_key: The access key, which should preferably be injected through an environment variable.

Do not write real keys directly into the configuration file. A safer approach is to set an environment variable:

export MODEL_API_KEY="replace-with-your-real-key"

To configure multiple models, add more entries:

model_list:
  - model_name: fast-coding
    provider:
      model: model-a
      api_base: https://service-a.example/v1
      api_key: ${MODEL_API_KEY_A}
  - model_name: deep-reasoning
    provider:
      model: model-b
      api_base: https://service-b.example/v1
      api_key: ${MODEL_API_KEY_B}

The client can then switch backends by changing the logical model name without modifying the endpoint or authentication logic.

6. Start the Gateway

Start the gateway with the configuration file:

gateway-server --config gateway-config.yaml --host 127.0.0.1 --port 4000

If the client and gateway run on the same device, listening on 127.0.0.1 is safer. To allow other devices on the local network to connect, use:

gateway-server --config gateway-config.yaml --host 0.0.0.0 --port 4000

Listening on all network interfaces increases the exposed attack surface. Use authentication, firewall rules, and network isolation as appropriate. Never expose the proxy port directly to the public internet without authentication.

After startup, check whether the port is listening:

ss -lntp | grep 4000

You can also test a local health endpoint:

curl http://127.0.0.1:4000/health

The exact health-check path depends on the gateway implementation. If that path is unavailable, test the model endpoint directly.

7. Verify the Model Request Path

The following is a generic message request:

curl http://127.0.0.1:4000/v1/messages \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-proxy" \
  -d '{
    "model": "coding-default",
    "max_tokens": 128,
    "messages": [
      {
        "role": "user",
        "content": "Explain what a recursive function is in one sentence."
      }
    ]
  }'

During testing, verify the following:

1. Whether a connection can be established;
2. Whether the server returns a valid HTTP status code;
3. Whether the response contains the model output;
4. Whether the gateway log shows an upstream request;
5. Whether the upstream service recognizes the model name;
6. Whether authentication information is forwarded correctly.

A successful response indicates that the basic client-to-gateway-to-model service path is working.

8. Connect the AI Coding Tool to the Gateway

Integration usually requires three types of configuration:

• The API base URL;
• The access key;
• The default model name.

For example:

export API_BASE_URL="http://127.0.0.1:4000"
export API_KEY="sk-proxy"
export MODEL_NAME="coding-default"

If the coding tool supports environment variables for its API endpoint, prefer this method. Environment variables make it easier to switch environments and reduce the need to modify application files.

When the gateway runs on another device on the local network, change the address:

export API_BASE_URL="http://192.168.1.20:4000"

Confirm that:

• Both devices are on the same local network;
• The gateway is not listening only on the loopback interface;
• The firewall allows the required port;
• The client can reach that port;
• The gateway has the necessary access authentication enabled.

9. Keep the Service Running

A service started directly in a terminal usually stops when the terminal closes. For long-term use, manage it with the system service manager.

Create a service file:

sudo nano /etc/systemd/system/model-gateway.service

Example contents:

[Unit]
Description=Local Model Gateway
After=network-online.target
[Service]
User=your-user
WorkingDirectory=/home/your-user/model-gateway
Environment=MODEL_API_KEY=replace-with-your-real-key
ExecStart=/home/your-user/model-gateway/.venv/bin/gateway-server --config /home/your-user/model-gateway/gateway-config.yaml --host 127.0.0.1 --port 4000
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target

Reload systemd and start the service:

sudo systemctl daemon-reload
sudo systemctl enable model-gateway
sudo systemctl start model-gateway

Check its status:

sudo systemctl status model-gateway

View the logs:

journalctl -u model-gateway -f

For production environments, consider adding log rotation, resource limits, automatic recovery, and disk-space monitoring. These controls help prevent long-running logs from consuming all available storage.

10. Security and Cost-Control Recommendations

A model gateway centralizes access credentials for multiple backends, so security requires careful attention.

1. Do Not Expose the Gateway Publicly

Unless there is a specific requirement, listen only on the local host or a trusted LAN address. Do not map the port directly to the public internet.

2. Use Environment Variables or Secret Management

Do not commit keys to a code repository or expose them in screenshots, logs, or shell history. You can use a permission-restricted environment file and apply suitable file permissions:

chmod 600 .env

3. Set Request Limits

Limit the maximum output length, number of concurrent requests, and timeout values. This helps prevent accidental overuse of resources.

4. Redact Sensitive Logs

Do not record complete keys, user privacy data, full source code, or sensitive business information in logs. Keep only necessary details such as request time, model identifier, status code, and latency.

5. Monitor Context Length

Different models may support different context capacities. When a project contains many files, use directory filtering, batched reads, and summary compression instead of sending excessive content in a single request.

11. Common Troubleshooting Issues

Issue 1: Connection refused
Check whether the gateway process is running, whether the port is correct, and whether the service is listening only on the local host.

Issue 2: Authentication failure
The credential used between the client and gateway is separate from the credential used between the gateway and the upstream service. Confirm that the two have not been mixed up.

Issue 3: Model not found
Check that the logical model name used by the client exactly matches the model_name value in the configuration file.

Issue 4: Request timeout
Possible causes include poor network quality, a slow upstream service, insufficient single-board computer resources, or a timeout value that is too short. Start with a minimal request to verify the route, then gradually increase the context size.

Issue 5: Response format cannot be parsed
The client and gateway may expect different protocols. Confirm that the gateway supports the message format required by the target client. If necessary, add a protocol conversion layer.

Conclusion

Deploying a model gateway on a low-cost single-board computer decouples an AI coding tool from any specific model service. The client handles interaction and code operations, the gateway manages routing, authentication, and protocol conversion, and backend models can be replaced according to the task.

The value of this design is not in making the single-board computer perform complex inference. Instead, it provides a stable, low-power, centrally managed model access node. After the basic deployment is complete, the system can be extended with multi-model routing, retries, access auditing, rate limiting, LAN sharing, monitoring, and alerting to create a more complete AI development infrastructure for individuals or small teams.