Deploying AI Inference Platform with Large Models on Ubuntu 24.04: A Complete Guide

    This guide provides step-by-step instructions for deploying an AI inference platform on Ubuntu 24.04 and integrating large language models, covering system requirements, environment setup, security configurations, and performance optimization.

    System Requirements and Environment Setup

    Hardware Recommendations

    • Minimum: 4-core CPU with 8GB RAM
    • Recommended for high concurrency: 16GB RAM
    • Disk space: ≥20GB for dependencies and model files

    System Initialization

    1. # Update system packages
    2. sudo apt update && sudo apt upgrade -y
    3. # Install development tools
    4. sudo apt install -y build-essential curl git
    5. # Configure firewall (open essential ports only)
    6. sudo ufw allow 22/tcp # SSH
    7. sudo ufw allow 3000/tcp # Default web port
    8. sudo ufw enable

    User Management

    1. # Create dedicated service user
    2. sudo adduser ai-service
    3. sudo usermod -aG sudo ai-service
    4. su - ai-service

    Node.js Environment Configuration

    Version Management with NVM

    1. # Install NVM
    2. curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.7/install.sh | bash
    3. source ~/.bashrc
    4. # Verify installation
    5. command -v nvm
    6. # Install LTS version
    7. nvm install 20
    8. nvm alias default 20
    9. node -v # Should show v20.x.x

    Version Management Best Practices

    • Production: Use even-numbered LTS versions (e.g., v20.x)
    • Development: Track latest versions
    • Project-specific versions: Use .nvmrc files

    AI Inference Platform Installation

    Global Installation

    1. # Install latest version
    2. npm install -g openclaw@latest --unsafe-perm
    3. # Verify installation
    4. openclaw --version

    Troubleshooting Common Issues

    Directory Conflicts:

    1. # Clean residual files
    2. rm -rf ~/.npm-global/lib/node_modules/openclaw
    3. npm cache clean --force
    4. # Reinstall
    5. npm install -g openclaw@latest

    Permission Errors:

    1. Fix ownership:
      1. sudo chown -R $USER ~/.npm
    2. Or configure non-root npm:
      1. mkdir ~/.npm-global
      2. npm config set prefix '~/.npm-global'
      3. echo 'export PATH=~/.npm-global/bin:$PATH' >> ~/.bashrc
      4. source ~/.bashrc

    Large Model Integration

    Model Service Configuration

    1. Obtain API key from model provider
    2. Verify service access permissions
    3. Record service endpoint URL

    Interactive Configuration

    1. openclaw config

    Follow prompts to configure:

    1. Provider type: Select “Custom Provider”
    2. Connection parameters:
      • Name: Custom identifier (e.g., zhipu-glm)
      • Endpoint: https://api.example.com/v4 (example)
      • API Key: Provider-issued authentication key

    Configuration Validation

    Check ~/.openclaw/config.json for:

    1. {
    2. "providers": {
    3. "zhipu-glm": {
    4. "type": "custom",
    5. "endpoint": "https://api.example.com/v4",
    6. "apiKey": "sk-xxxxxxxxxxxxxxxx"
    7. }
    8. }
    9. }

    Service Deployment and Security

    Development Server

    1. openclaw serve --port 3000

    Verify with:

    1. curl http://localhost:3000/health
    2. # Should return {"status":"ok"}

    Production Deployment

    1. Process management with PM2:

      1. npm install -g pm2
      2. pm2 start `which openclaw` --name ai-service -- serve --port 3000
      3. pm2 save
      4. pm2 startup
    2. Nginx reverse proxy configuration:

      1. server {
      2. listen 80;
      3. server_name ai.example.com;
      4. location / {
      5. proxy_pass http://localhost:3000;
      6. proxy_set_header Host $host;
      7. proxy_set_header X-Real-IP $remote_addr;
      8. }
      9. }

    Secure Access Options

    SSH Tunneling:

    1. # Local tunnel setup
    2. ssh -L 3000:localhost:3000 user@server-ip
    3. # Browser access
    4. http://localhost:3000

    VPN Access:

    1. Deploy WireGuard VPN
    2. Configure firewall to allow 3000 only from VPN subnet
    3. Access services through VPN connection

    Performance Optimization

    Monitoring Setup

    1. # Install monitoring tools
    2. sudo apt install -y htop sysstat
    3. # Configure Node.js monitoring
    4. npm install -g pm2-logrotate
    5. pm2 set pm2-logrotate:max_size 10M

    Performance Tuning

    1. Adjust Node.js memory:

      1. export NODE_OPTIONS="--max-old-space-size=4096"
    2. Implement rate limiting:

      1. {
      2. "rateLimit": {
      3. "max": 100,
      4. "windowMs": 60000
      5. }
      6. }

    Troubleshooting Guide

    Common Error Codes

    Code Cause Solution
    502 Service down Check service status/logs
    403 Auth failed Verify API key validity
    504 Timeout Adjust model timeout settings

    Log Analysis

    1. Service logs:

      1. pm2 logs ai-service
    2. Enable debug logging:

      1. export DEBUG=openclaw:*
      2. openclaw serve
    3. System logs:

      1. journalctl -u nginx --no-pager -n 100

    Advanced Features

    Multi-Model Support

    Update configuration:

    1. {
    2. "providers": {
    3. "model-a": {...},
    4. "model-b": {...}
    5. }
    6. }

    Custom Routing

    Create routes.js:

    1. module.exports = (app) => {
    2. app.get('/custom', (req, res) => {
    3. res.send('Custom endpoint');
    4. });
    5. };

    Start with:

    1. openclaw serve --routes ./routes.js

    Authentication Integration

    1. Install Passport.js:
      1. npm install passport passport-jwt
    2. Configure JWT middleware

    Maintenance and Upgrades

    Version Upgrades

    1. # Check for updates
    2. npm outdated -g openclaw
    3. # Perform upgrade
    4. npm update -g openclaw
    5. # Verify version
    6. openclaw --version

    Backup Strategy

    1. Configuration backup:

      1. tar czvf openclaw-config-$(date +%Y%m%d).tar.gz ~/.openclaw/
    2. Database backup (if applicable):
      ```bash

      Locate database file

      find / -name “openclaw.db” 2>/dev/null

    Perform backup

    cp /path/to/openclaw.db /backups/
    ```

    Best Practices Summary

    1. Security First: Always access admin interfaces through VPN/SSH tunnels
    2. Resource Isolation: Create separate users for different model services
    3. Monitoring: Configure alerts for critical metrics
    4. Maintenance: Establish weekly maintenance windows
    5. Documentation: Maintain detailed deployment records

    This standardized approach enables developers to quickly deploy secure, production-ready AI inference platforms on Ubuntu 24.04 while maintaining flexibility for customization. The solution balances development convenience with production stability requirements.