Scaling LLM Applications: Compute Challenges and Plugin-Based Agent Architectures

    This article examines the engineering challenges behind high-concurrency large language model applications, including inference optimization, queue-based traffic smoothing, capacity planning, autoscaling, plugin governance, and agent architecture. It also provides a RabbitMQ reference implementation, a tool schema example, and practical guidance for building safer, more reliable production systems.

    Public reports indicate that one general-purpose large language model application has exceeded one million daily active users, consumed tens of trillions of tokens per day, supported tens of thousands of third-party plugins, and released an open-source super-agent research prototype. These figures show that LLM applications are moving from algorithm validation to large-scale engineering.

    The figures above come from public reports. Their statistical periods and token definitions—such as request counts, input tokens, or combined input and output tokens—have not been published under a unified standard. Official disclosures should therefore be used when citing them.

    This article focuses on three areas: high-concurrency inference, plugin ecosystems, and agent architecture.

    1. Compute and Architecture Challenges in High-Concurrency Inference

    A system serving millions of daily users and processing tens of trillions of tokens per day cannot solve throughput bottlenecks simply by adding more GPUs. Optimization is required across the inference engine, traffic scheduling, and elastic scaling layers.

    1.1 Inference Engine Optimization

    KV cache usually consumes the largest share of GPU memory and is a primary bottleneck in high-concurrency scenarios. PagedAttention manages the KV cache as fixed-size blocks, reducing memory fragmentation and allowing requests with the same prefix to share cache blocks. This increases the number of concurrent requests that can be supported.

    Continuous batching dynamically adds new requests and removes completed requests at each decoding step. This prevents GPUs from sitting idle when short and long requests are processed together.

    These techniques have different applicability boundaries. PagedAttention requires additional block-table management and inference-kernel support. Continuous batching is particularly useful when request lengths vary significantly, but provides limited benefit for offline workloads in which request lengths are highly consistent. Before selecting an approach, benchmark it against your own request distribution.

    1.2 Using a Message Queue to Smooth Traffic Spikes

    Traffic spikes can cause synchronous calls to time out, which may trigger cascading failures. A message queue decouples request intake from model inference. The following example shows the basic producer-consumer structure. It requires pika 1.x; install it with pip install pika.

    This is a reference implementation. The caller must implement run_inference and connect it to the relevant inference engine.

    1. import json
    2. import os
    3. import pika
    4. AMQP_URL = os.environ["AMQP_URL"] # Read from an environment variable; do not hard-code it
    5. QUEUE = "llm_inference_tasks"
    6. def publish_inference_request(user_id: str, prompt: str) -> None:
    7. connection = pika.BlockingConnection(pika.URLParameters(AMQP_URL))
    8. try:
    9. channel = connection.channel()
    10. channel.queue_declare(queue=QUEUE, durable=True)
    11. body = json.dumps({"user_id": user_id, "prompt": prompt}, ensure_ascii=False)
    12. channel.basic_publish(
    13. exchange="",
    14. routing_key=QUEUE,
    15. body=body,
    16. properties=pika.BasicProperties(delivery_mode=2), # Make the message persistent
    17. )
    18. finally:
    19. connection.close()
    20. def handle_task(channel, method, properties, body):
    21. try:
    22. task = json.loads(body)
    23. run_inference(task) # Implement this and connect it to the inference engine
    24. channel.basic_ack(delivery_tag=method.delivery_tag)
    25. except Exception:
    26. # Do not requeue failed messages indefinitely; route them to a dead-letter queue
    27. channel.basic_nack(delivery_tag=method.delivery_tag, requeue=False)
    28. def consume() -> None:
    29. connection = pika.BlockingConnection(pika.URLParameters(AMQP_URL))
    30. channel = connection.channel()
    31. channel.queue_declare(queue=QUEUE, durable=True)
    32. channel.basic_qos(prefetch_count=1) # Each consumer processes only one task at a time
    33. channel.basic_consume(queue=QUEUE, on_message_callback=handle_task)
    34. try:
    35. channel.start_consuming()
    36. finally:
    37. connection.close()

    Several design details are important:

    • Queue durability and message persistence must be enabled together to preserve tasks after a broker restart.
    • Setting prefetch_count to 1 prevents a single consumer from accumulating too many tasks.
    • Failed messages should be routed to a dead-letter queue. Otherwise, they must either be discarded or allowed to accumulate.

    1.3 Capacity Planning and Load-Testing Checklist

    • Peak QPS: Divide the daily request volume by 86,400, then multiply it by a peak factor. A factor of 3 to 5 is commonly used as an initial estimate, but it must be validated against actual traffic data.
    • Number of nodes: Divide peak tokens per second by the throughput of one tested instance multiplied by the target utilization. A target utilization of 0.6 to 0.7 is recommended to leave room for traffic bursts.
    • Load-testing metrics: Measure time to first token (TTFT), time per output token (TPOT), P99 latency, queue wait time, GPU memory usage, and request failure rate.
    • Degradation strategies: When thresholds are exceeded, switch to a smaller model, reduce the maximum output length, or return a clear queueing message.
    • Rate limiting: Apply token-bucket limits by user and tenant to prevent one user from consuming all inference capacity.
    • Alerting: Monitor queue wait time when it exceeds a threshold, such as 30 seconds; continuous growth in the dead-letter queue; and P99 timeout rates above 1%.

    1.4 Elastic Scaling

    Inference nodes should scale according to queue length and wait time. Because loading a large model can take several minutes, scaling policies must account for model startup time. Maintain a minimum number of replicas so that capacity is available when traffic rises again and newly started nodes are still initializing.

    2. Plugin Ecosystems: From Rapid Growth to Standardization

    Function calling allows a model to select an appropriate tool based on user intent and generate structured parameters. The backend executes the tool, returns the result to the model, and lets the model organize the result into a natural-language response. Tool descriptions commonly use JSON Schema. The following example defines a weather lookup tool.

    1. {
    2. "name": "get_weather",
    3. "description": "Get the current weather for a specified city",
    4. "parameters": {
    5. "type": "object",
    6. "properties": {
    7. "city": {
    8. "type": "string",
    9. "description": "City name, for example, Beijing"
    10. },
    11. "unit": {
    12. "type": "string",
    13. "enum": ["celsius", "fahrenheit"],
    14. "description": "Temperature unit"
    15. }
    16. },
    17. "required": ["city"]
    18. }
    19. }

    As the number of third-party plugins grows, quality becomes inconsistent and security risks become more common. An official plugin marketplace can provide standardized interface definitions, security reviews, authentication, and billing mechanisms. Developers gain a trusted distribution channel, while enterprise users can reduce the compliance risks of integrating external capabilities.

    Plugin Integration Checklist

    • Least-privilege access: Request only the permissions required for the tool and let users approve permissions individually.
    • Input validation: Validate the type, length, and allowed values of model-generated parameters. Never concatenate them directly into SQL statements or shell commands.
    • Timeouts and retries: Set a timeout for every call. Retry only idempotent operations and use exponential backoff.
    • Audit logs: Record the caller, parameter summary, return status, and duration. Redact sensitive fields before persisting logs.
    • Billing and quotas: Measure usage by call count or token consumption and set per-user limits.
    • Data boundaries: Explain what data the plugin can read and store, and allow users to view and delete that data.
    • Prompt-injection protection: Tool responses may contain malicious instructions. Treat tool output as data rather than executable instructions.

    3. Super-Agent Research Prototypes: Concepts and Components

    An agent is a system that can perceive its environment, plan tasks, retain memory, and call tools. Unlike a one-turn question-and-answer system, it can execute multiple steps toward a goal. A super-agent research prototype is an open-source framework intended for research. Additional access controls, monitoring, and fault-tolerance mechanisms are required before deploying such a system in production.

    A complete agent typically contains three categories of modules:

    • Planning module: Breaks complex tasks into subtasks. The ReAct (Reasoning and Acting) framework alternates between reasoning and action and is a common implementation pattern.
    • Memory module: Short-term memory relies on the context window, while long-term memory commonly uses a vector database for semantic retrieval.
    • Action module: Performs concrete operations by calling a code interpreter or external plugins.

    Common Risks

    • Long-term memory retrieval failures: Retrieved content may be irrelevant or outdated, resulting in incorrect decisions. Retrieval similarity scores should be recorded, with a minimum threshold applied.
    • Unauthorized tool calls: An agent may call tools outside the scope of its task. Use a tool allowlist, and require human confirmation for high-risk operations such as deletion, transfers, or sending messages.
    • Planning hallucinations: A model may invent nonexistent steps or tool names. Validate tool names and parameters, return to the planning stage after failures, and set a maximum number of execution steps to prevent infinite loops.

    Infrastructure Selection

    Production agents require several types of infrastructure. Long-term memory can be implemented with a vector database or object storage. Runtime logs should be integrated with a logging service for end-to-end tracing. The execution state of multi-step tasks is typically stored in a distributed cache.

    Cloud providers offer cloud databases, container orchestration, logging services, and vector search services that can fulfill these roles. When comparing options, consider read and write latency, storage cost per unit, index types, multi-tenant isolation, and backup and recovery capabilities.

    4. Summary and Implementation Recommendations

    The engineering focus of LLM applications is shifting from the model itself toward system design.

    A practical implementation sequence is:

    1. Estimate capacity and conduct load tests.
    2. Identify whether the bottleneck is GPU memory, queues, or scheduling.
    3. Select inference optimizations based on measured workloads.
    4. Establish authentication, auditing, and input validation before expanding plugin capabilities.
    5. Run agents with a restricted tool set first.
    6. Gradually expand permissions only after behavior has proven stable.

    This approach helps teams address scalability, plugin security, and agent reliability in a controlled manner.