Building Mobile AI Chat Apps: Streaming, Context Management, and Cost Control

1. Background: What Makes Mobile AI Chat Hard

An LLM-powered mobile chat app runs on the phone but relies on cloud inference for text generation, multi-turn Q&A, and content creation. The constraints on the mobile side are hard: limited compute, a network that keeps switching between Wi-Fi and cellular, and users with zero tolerance for long waits.

Developers usually ask the same handful of questions before they start building: how do you implement SSE correctly on mobile, what do you do when the context window overflows, how do you pick among model tiers, how do you handle a weak or absent connection, and how do you keep token costs down? This article answers them in order: architecture, protocol, streaming, context, security, performance, and acceptance.

2. Layered Architecture and Protocol Choice

A typical architecture has three layers:

• Client layer: UI rendering, input capture, message history management, incremental streaming display, local caching, and offline fallback.
• Gateway / BFF layer: authentication, request validation, rate limiting and quotas, logging and instrumentation, protocol translation, and safe forwarding to the model service.
• Model service layer: the actual inference responsible for completion and generation.

The core idea is a thin client and a heavy backend: the mobile app never runs the model, it only handles interaction and state management.

Protocol choice is the key decision at this layer. The tradeoffs on mobile networks:

Protocol	Best fit	Main tradeoffs
SSE (over HTTPS)	Default for one-way streamed text output	On HTTP/1.1 it is subject to the 6-connection-per-origin limit (HTTP/2 multiplexing mitigates this); the standard Last-Event-ID reconnection does not carry a custom request body, so resumption must be implemented yourself; text frames only
WebSocket	Bidirectional low latency, e.g. voice interruption, real-time collaboration	Heartbeats, reconnection, timeouts, and message IDs are all on you; handshake failures are more common on some mobile networks and behind enterprise proxies; harder to reuse the HTTP auth and caching chain
HTTP long polling	Very old WebViews or restricted proxies	Many idle requests, high latency, battery drain; fallback only

For most pure-text chat scenarios, SSE is the choice: cheap to implement, a natural fit for push-as-you-generate, and the best compatibility through proxy devices.

3. Model Tiers and Upgrade / Fallback Strategy

By parameter count and capability boundary, models are commonly grouped into three tiers:

• Lightweight model: fast responses, low unit price; suited to simple Q&A, rewriting, and summarization; the default tier under high concurrency.
• Balanced model: a balance of capability and efficiency; covers most general conversation and creation needs.
• High-capability model: complex reasoning, code generation, long-context handling; higher latency and cost.

Recommended approach: tiered routing. Use the lightweight model to return the first token fast, and escalate when a complex request is detected. But escalation raises a new question: once the high-capability model’s answer arrives, what happens to the lightweight content already streamed to the screen? Three strategies:

• Regenerate: the lightweight content is only a draft and is replaced wholesale when the high-capability result arrives. The client must clear the draft stream rather than concatenate the two.
• Continuation: when the lightweight answer is already complete and accepted by the user, the high-capability model only fills the gaps, marked with an explicit separator.
• The deduplication rule: the server assigns a requestId per conversation turn; when the client receives a new requestId it discards all subsequent deltas from the old stream, preventing double writes.

Write the fallback conditions into code explicitly: fall back to the balanced model when the high-capability model’s first token exceeds a threshold (e.g. 2.5s), when it returns 429, when the error rate exceeds a threshold, or when the daily cost budget is exhausted. If the balanced model is also unavailable, show the lightweight draft labeled “full answer generating.”

4. Streaming Responses: A Working SSE Parser

LLMs generate token by token, so waiting for the complete answer means a long blank screen. Streaming output is effectively mandatory. The code below performs real SSE parsing: it splits on event boundaries (blank lines), extracts the data field, handles the [DONE] terminator, and tolerates a TCP segment containing half an event or several events at once.

async function streamChat(messages, { onDelta, signal } = {}) {
  const response = await fetch("https://your-gateway.example/api/chat", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "Authorization": `Bearer ${token}`
    },
    body: JSON.stringify({ messages, stream: true }),
    signal                       // supplied by an external AbortController, for cancellation
  });
  if (!response.ok || !response.body) {
    throw new Error(`Chat request failed: HTTP ${response.status}`);
  }
  const reader = response.body.getReader();
  const decoder = new TextDecoder();
  const boundary = /\r\n\r\n|\n\n|\r\r/;   // SSE event boundary, tolerant of several line-ending styles
  let buffer = "";                        // leftover data that does not yet form a complete event
  let answer = "";                        // confirmed body text, preserved if interrupted
  while (true) {
    const { done, value } = await reader.read();
    if (done) break;
    buffer += decoder.decode(value, { stream: true });
    let match;
    while ((match = boundary.exec(buffer)) !== null) {
      const rawEvent = buffer.slice(0, match.index);
      buffer = buffer.slice(match.index + match[0].length); // keep the partial packet for the next round
      // One event may contain multiple data: lines; per spec they are joined with newlines
      const dataLines = [];
      for (const line of rawEvent.split(/\r\n|\n|\r/)) {
        if (line.startsWith(":")) continue;          // comment / heartbeat, e.g. ":ping"
        if (line.startsWith("data:")) dataLines.push(line.slice(5).trimStart());
      }
      if (dataLines.length === 0) continue;
      const payload = dataLines.join("\n");
      if (payload === "[DONE]") return answer;       // server-agreed terminator
      let chunk;
      try {
        chunk = JSON.parse(payload);
      } catch {
        console.warn("Discarding unparseable SSE fragment");
        continue;                                    // bad data must not abort the whole conversation
      }
      const delta = chunk.choices?.[0]?.delta?.content ?? "";
      if (delta) {
        answer += delta;
        onDelta?.(answer);        // pass the cumulative text, so the renderer does not concatenate itself
      }
    }
  }
  return answer;
}

Cancellation, Reconnect, and Deduplication

Cancelling generation uses AbortController, and generated content must not be lost:

const controller = new AbortController();
let partial = "";
try {
  await streamChat(messages, {
    signal: controller.signal,
    onDelta: (text) => { partial = text; renderIncremental(text); }
  });
} catch (err) {
  if (err.name === "AbortError") {
    // The user stopped it on purpose: keep the generated content, mark the state, avoid billing disputes
    renderIncremental(partial, { interrupted: true });
    return;
  }
  await reconnectOrDegrade(messages, partial);   // network failure goes to reconnect / degradation
}

Four fault-tolerance rules for the streaming path:

1. Reconnect on disconnect: the server attaches an incrementing seq to every chunk and the client records the largest seq displayed; on reconnect it skips chunks with seq <= maxSeq, giving idempotent resumption. If the server does not support seq, degrade to whole-segment replacement instead of appending, or compare by the length of the received prefix.
2. Deduplicating repeated tokens: retries inevitably produce duplicate pushes, so deduplication must be done by sequence number or prefix comparison, never by “it looks about the same.”
3. Preserving generated content: no exception may clear the text on screen; only append a status notice.
4. Error fallback and heartbeats: switching from Wi-Fi to 4G can silently kill a long-lived connection, so you need a client-side read timeout plus server-sent SSE comment heartbeats; on timeout show “Network disconnected, tap to retry” and keep the input box contents.

5. Context Management and Token Budget

Models do not “remember.” Multi-turn conversation works by sending the history back with every request. Messages typically use the three roles system / user / assistant:

{
  "messages": [
    { "role": "system", "content": "You are a helpful AI assistant" },
    { "role": "user", "content": "Help me summarize the key points of this article" },
    { "role": "assistant", "content": "Sure, here are the key points..." },
    { "role": "user", "content": "Can you expand on the first point?" }
  ]
}

Calculating the Token Budget

Take an 8K context window: a fixed system prompt plus tool descriptions costs about 600 tokens, the current user input about 200 tokens, and you reserve 800 tokens for the reply, so the history budget is 8000 - 600 - 200 - 800 = 6400 tokens. For Chinese text you can roughly estimate 1 Chinese character ≈ 0.6–1 token, but always measure against the target model’s tokenizer. In practice, set the trigger line at 80% of the budget (about 5100 tokens) to leave room for variance.

Rolling Summary Template and Key-Fact Retention

When the limit is exceeded, compress earlier messages into a summary and keep only the recent turns plus the summary. The summary template has fixed fields to prevent free-form drift:

[User goal]       {what this conversation is trying to achieve}
[Confirmed facts] {fact 1; fact 2; fact 3}
[Preferences]     {language, format, length, prohibitions}
[Open questions]  {what has not been resolved yet}

Key facts never go into the summary: names, project names, amounts, dates, and constraints the user stated explicitly are stored as separate structured fields and sent back verbatim on every turn. This is where rolling summaries lose information most easily — once “budget of 30,000” gets compressed into “discussed the budget,” every later answer will be wrong.

6. Security, Privacy, and Compliance

Beyond the baseline requirements — transport encryption, authentication and rate limiting, content moderation, and keeping keys server-side only — logs, caches, and summaries are the three places that most easily become privacy holes, so define explicit rules:

• Minimize collection: prompts and model replies are not logged by default; troubleshooting needs only requestId, model tier, token count, latency, and status code. When samples really are needed, store only length and a hash.
• Field masking: phone numbers, ID numbers, emails, and card numbers are detected through both regex and NER, then masked before they enter logs, caches, or summaries — not obfuscated at display time.
• Encrypted storage and retention: encrypt data at rest with keys managed by a KMS; keep debug logs for 7 days, audit logs for 180 days, and do not retain conversation text by default. When a user enables “history,” provide export and one-tap deletion, and make deletion also clear summaries and caches.
• User consent: state the purpose of data use clearly on first use, and offer a separate, off-by-default toggle for “use data to improve the model.”
• Cache boundaries: cache only publicly shareable, non-personalized content (prompt templates, general knowledge answers, static copy). Never cache responses containing user identity or session context. Cache keys must include model version, prompt version, and key parameters; a TTL of 5–30 minutes is recommended, with support for per-user deletion.
• Credential isolation: call keys live only on the server, never hardcoded in the mobile app or written to local storage.

7. Performance, Cost, and Observability

1. First-token latency: streaming prioritizes returning the first token so users perceive a response as early as possible.
2. Tiered routing: pick a tier based on task complexity to balance quality against cost.
3. Cache reuse: cache only the “publicly shareable, non-personalized” content described above, with versioned keys and a TTL.
4. Server-side inference batching: the gateway can apply dynamic batching / continuous batching to concurrent requests, combining multiple users’ requests into one batch to raise GPU utilization. Note that this is not the same as “merging several requests from one user” — a single user’s single chat request cannot be split and merged, end-to-end latency will not improve, and poor configuration will actually increase per-request queueing time.
5. Offline fallback: handle three scenarios separately.

Scenario	Detection	Handling strategy
Weak network	Reachable but high RTT, first token over threshold	Drop to the lightweight model, compress context, show a skeleton screen, disable extra capabilities
No network	Probe fails or no connectivity	Show locally cached sessions and drafts, queue input locally, and ask whether to send after recovery
Service failure	5xx, timeout, 429	Retry with exponential backoff, at most 2 attempts and only for idempotent requests; if it still fails, fall back to a backup model or return an explicit error code

Metrics to monitor: first-token latency (TTFT) P50/P95, end-to-end completion latency P95, error rate broken down by error code, user cancellation rate, cache hit rate, token consumption and derived cost per session, context truncation trigger rate, and content moderation block rate. Example alert thresholds: TTFT P95 above 2s for 5 minutes, error rate above 2%, 429 share above 5%, daily cost above 80% of budget. Optimization without a measurement definition cannot be verified, and “it feels faster” is not a delivery criterion.

8. Rollout Path and Acceptance Checklist

Start from a minimum viable version: get the main path working first — authentication, request, streaming response, incremental rendering — then introduce context summarization, tiered routing, content safety, and batching step by step.

Verify each item before launch:

• Keys are stored server-side only, with no plaintext credentials on the client
• SSE parsing correctly handles cross-segment events, multiple data lines, [DONE], and invalid JSON
• Cancellation works, generated content is preserved, and the state is labeled correctly
• Reconnect deduplicates by seq, with no duplicated text
• All three degradation paths — weak network, no network, service failure — can be triggered
• Logs and caches are masked, and retention periods and deletion work
• Bidirectional content moderation is wired up for input and output
• Context overflow triggers summarization, and key fact fields are not lost
• Tier escalation produces no duplicate output, and fallback conditions are observable
• TTFT P95, error rate, and token cost are connected to monitoring and alerting

A mobile AI chat app is, at its core, the engineering fusion of “cloud model capability” and “mobile interaction experience.” The tradeoffs that need continuous attention are three pairs: experience vs. cost, capability vs. latency, and feature richness vs. device constraints. Get the four fundamentals right — layered architecture, the streaming protocol, context management, and security — and the remaining feature work is only a matter of time.