Building Large-Scale User Interview Systems with Large Language Models

As AI becomes widely used, enterprises and research groups can already infer usage patterns and economic trends from logs and behavioral instrumentation. Privacy-preserving analytics can even study large volumes of interaction traces safely. Those methods still have a hard limit: they show what users do, not how they apply model outputs after a session, how they actually felt about the interaction, or what role they imagine AI should play in future daily life.

To understand AI’s evolving place in society and to involve users in development and alignment, the most direct path is to ask them. Running hundreds or thousands of in-depth human interviews, however, is prohibitive in time, staff, and cost. A practical response is to use the conversational ability of large language models themselves to run an automated interview system at scale. This article covers the architecture, key implementation details, and the feedback loop from interviews into research and product work.

The Gap from Behavior to Deeper Insight

Traditional product analytics surface structured metrics: DAU, average turns, token consumption, and similar. Combined with privacy-preserving tools, they sketch a high-level picture of economic use. Metrics lack context.

A log may show that a user asked for code; it does not show whether that code went to production or was only used for learning. A long emotional conversation may be recorded without any signal of whether the user felt better or worse afterward. The jump from behavior to intent to impact is one that logs cannot make. Empathetic, exploratory interviews are the practical way to close that gap.

Why Traditional Interviews Do Not Scale

Qualitative interviews typically rely on experienced researchers using semi-structured prompts. They yield dense context but hit three constraints:

1. Low throughput. A deep session often lasts 45–60 minutes plus prep and write-up, so daily volume stays small.
2. Hard-to-control bias. Interviewer style and follow-up depth vary, which makes comparison and quantification difficult.
3. Limited sample diversity. Geography, time zones, and language restrict who can be reached.

Core Architecture: An LLM-Based Automated Interview System

To address those limits, the team built an LLM-based automated interview stack. It runs deep conversations with users at unprecedented scale and returns de-identified transcripts for human researchers to analyze at a macro level.

The design has three layers:

1. Trigger and scheduling. Privacy-preserving analytics identify users with representative usage patterns. After a session ends, an invitation is enqueued asynchronously.
2. AI interview engine. This is the core. Given light context (for example, recent feature categories), the LLM generates questions and follows up over multiple turns based on answers.
3. Aggregation and analysis. Interview text is persisted in object storage. NLP extracts keywords, sentiment, and core needs into structured reports for researchers.

Key Implementation and Practices

Several engineering dimensions matter when building this kind of system.

1. Context-Aware Dynamic Follow-Ups

Unlike static surveys, depth comes from follow-up. The model must steer based on what the interviewee just said. That is typically done with a carefully written system prompt and conversation-window management.

A simplified interview-engine loop:

def generate_interview_question(user_context, conversation_history, user_response):
    system_prompt = """
    You are a senior UX researcher. Your task is to conduct an in-depth interview about the user's experience with an AI assistant.
    Follow these principles:
    1. Stay neutral and empathetic; do not lead the user.
    2. Ask only one question at a time.
    3. If the answer is vague, probe a concrete detail (e.g., "You mentioned using the result at work—can you describe the scenario?").
    4. Focus on: how AI output is actually used, emotional reactions, and expectations for the future.
    """
    messages = [
        {"role": "system", "content": system_prompt},
        {"role": "system", "content": f"User context summary: {user_context}"},
        *conversation_history,
        {"role": "user", "content": user_response if user_response else "Start the interview. Ask the first question."}
    ]
    # Call the LLM API (a general LLM service such as Baidu AI Cloud Qianfan can be used)
    next_question = llm_client.chat_completion(messages=messages, temperature=0.7)
    return next_question

A moderate temperature (for example 0.7) keeps reasoning coherent while allowing some flexibility and a more natural tone.

2. Privacy and De-identification

At scale, users often leak PII in conversation. Records must be redacted before they reach analysis. Text-moderation APIs or a combination of regex and NER can mask phone numbers, ID numbers, real names, and addresses in near real time.

3. Concurrent Scale and Cloud-Native Support

Hundreds or thousands of concurrent interviews need elastic, highly available infrastructure. In practice, cloud-native components (as used with Baidu AI Cloud) can back the service:

• Message queues absorb bursts of interview triggers so the system stays stable under load.
• Container instances (for example CCR/CCE) host stateless interview-scheduling microservices with HPA based on queue depth.
• Object storage and logging archive raw interviews as JSONL and stream operational logs so teams can monitor quality and dropped sessions.

Closing the Loop: From Interviews Back to Model Development

After tens of thousands of interviews, researchers receive not raw transcripts but pre-structured datasets.

Clustering can group “actual use cases” (code assistance, copy editing, emotional support, and so on). Sentiment models can quantify satisfaction by scenario. Together they form a richer picture of socio-economic use.

Those insights feed research and product teams:

1. Alignment. If many users are confused in a given scenario, instruction-following in that domain can be improved.
2. Risk discovery. Interviews can surface how users feel when they try edge-case uses, giving safety teams an earlier signal.
3. Product direction. Stated expectations for future AI help prioritize the next generation of features.

Conclusion

Using AI to understand human needs around AI is both a technical necessity and a path toward more harmonious human–machine systems. An LLM-based large-scale interview system can deliver deep qualitative insight at low marginal cost. Scaling qualitative research and turning subjective experience into data strengthens alignment and iteration, and keeps development closer to a human-centered goal.