Scientific LLM Usage Distribution: Data Insights and Lessons for Building Scientific AI Tools

1. Research Background and Data Sources

A recent public observation report on AI-assisted scientific research has drawn attention. When discussing the real boundaries of AI-assisted research and scientific LLMs, the value lies not in promoting a product, but in quantifying how researchers actually use AI assistants across disciplines, task types, and usage frequencies by analyzing large-scale real conversation samples.

Source and Scope Notes (Important)

This report is an observation report on scientific research usage published by a single AI platform. It collected approximately 207,000 research-related conversations, including about 129,000 general dialogue and collaborative office-work conversations and about 78,000 coding-related conversations.

When citing its data, readers should rely on the official report and verify the following information: the full report title, the publisher, the publication date and link, the statistical window, and the sample scope. It must be emphasized that the data comes from a single platform, so the conclusions cannot be directly extrapolated to all AI tools or to the entire scientific research population. In addition, the report’s baseline for “actual time invested by researchers” typically comes from self-reported questionnaires or platform-side timing. Different measurement methods can affect the credibility of subsequent comparisons, so the data source should be noted during interpretation.

The report’s data governance approach is worth learning from. During data collection, only classification labels such as the discipline, research subfield, and task type are retained. Specific research data, hypotheses, experimental methods, or interim findings are not extracted or stored. All conclusions are presented in aggregated statistical form. This provides a compliance example for research-oriented AI tools: it balances “analyzing usage patterns” with “not accessing research results,” usually requiring two levels of treatment:

• Labeling: Classification is completed before data storage, and only metadata is retained.
• Aggregation: Only statistical results are exposed externally, while identifiable information about individuals or single conversations is masked.

2. Discipline Distribution: Structural Differences in AI Usage

From a discipline perspective, the distribution of scientific AI conversations shows clear structural characteristics. Taking conversational sessions as an example, computer science accounts for the highest share (about 11.9%), engineering accounts for about 8.4%, and medicine accounts for about 7.7%. Disciplines such as medical health and physics, which often rely heavily on laboratory operations, show relatively low usage shares (mostly below 5%).

It should be noted that “medicine” refers to a discipline label, while “medical health” may refer to conversation classification by application field. Whether the two are equivalent depends on the platform’s definition, and they should not be used interchangeably.

Coding-related tools are further concentrated in quantitative research fields:

Researcher Category	Share of Global Researchers	Share of Conversational Sessions	Share of Coding Sessions
Computer science researchers	About 4.3%	Not separately disclosed	About 32.7%
AI/ML researchers	About 1.2%	About 5%	About 15.2%

Note: The original report did not separately disclose the conversational-session share for computer science researchers, so it is left blank here.

This table reveals a phenomenon: usage is not simply proportional to the population size of a discipline. AI/ML researchers account for only about 1.2% of global researchers but contribute about 15.2% of coding-related conversations. A reasonable explanation is that usage density is related to whether research tasks are suitable for AI intervention, rather than being determined solely by base population size. The more a task can be coded, textualized, or standardized, the deeper AI penetration tends to be. However, this is only one explanation based on task attributes. Discipline-level differences may also be affected by factors such as tool availability and platform distribution, so more data is needed for confirmation.

3. Task-Level Comparison: High and Low Usage Density

A valuable part of the report is its breakdown of task types. The table below lists the share of each task under different denominators:

Task Type	Share of Conversational AI Sessions	Share of Coding AI Sessions	Share of Researcher Time
Data analysis	About 33.7%	About 48.1%	About 27.6%
Writing and communication	About 32%	Not separately disclosed	About 16%
Data collection	About 11.5%	About 11.9%	About 28.6%
Research ideation and hypothesis generation	About 4.2%	About 2.1%	About 7.1%

Important note: The denominators for the three types of shares above are different (total conversational sessions, total coding sessions, and total researcher time). They cannot be directly compared, and they do not represent the AI replacement rate for a task. Based on this, several cautious conclusions can be drawn:

1. Data analysis and writing have high usage density. Data analysis accounts for about half of coding-related conversations, while writing and communication account for about one-third of conversational sessions, indicating that they are currently high-frequency usage scenarios. However, proving that they are “must-have” needs requires further evaluation combined with time savings and output quality.

2. Data collection has relatively low usage density. It accounts for about 28.6% of researcher time but only about 11.5%–11.9% of AI conversations. One possible reason is that data collection often depends on experimental equipment, on-site operations, field studies, or collection processes constrained by ethical approval. These steps are difficult for AI in a purely software form to replace directly.

3. Hypothesis generation has the lowest penetration. Research ideation and hypothesis generation account for only about 2.1%–4.2% of AI conversations and about 7.1% of researcher time. It should be noted that the denominators for time share and AI conversation share are different. If tasks are roughly estimated by task combination, data analysis and writing together account for about 65.7% of conversational sessions, which is higher than their combined time share of about 43.6%. This is because a large number of discipline-specific tasks are folded into these two categories. Therefore, the difference between shares cannot be simply treated as the capability boundary of AI; it is more a result of how tasks are classified.

4. Why These Differences Occur: Three Technical Dimensions

Abstracting the above phenomena, three technical dimensions determine whether AI is suitable for a scientific research task:

• Digitizability: Whether inputs and outputs can be represented as text, numbers, or structured data. For data analysis, inputs are tables and numerical values, and outputs are statistical conclusions, making it highly digitizable. Operations on the experimental bench cannot be digitized, so AI finds it difficult to intervene.
• Verifiability: Whether results are easy to validate automatically or at low cost. Correlations and charts can be verified through recomputation, and the cost of errors is controllable. In contrast, the correctness of a hypothesis may require years of experiments to falsify, so AI cannot guarantee quality at generation time.
• Creative requirement: Whether original thinking that breaks through existing paradigms is required. The more a task relies on experience, intuition, and domain depth, such as hypothesis generation, the more obvious the limitations of AI as a “probabilistic predictor based on historical patterns.”

# Illustration: AI suitability scoring framework for scientific research tasks
# Note: The three dimensions are combined linearly using weights, not hard constraints.
# Digitizability and verifiability are positive scoring factors; a higher creative requirement lowers the weight (inverse constraint).
# Verifiability is handled bidirectionally: easy recomputation adds points, while requiring long-term experiments to falsify adds an additional penalty.
# Thresholds are tunable parameters and should be calibrated per scenario; default values and a fallback strategy are shown below.
def assess_ai_suitability(task, w_digi=0.4, w_veri=0.4, w_crea=0.2,
                          high_threshold=0.6, low_threshold=0.35):
    """Return the task's AI suitability score (0-1) across the three dimensions.
    A higher score indicates greater suitability for AI assistance."""
    dimensions = {
        "data_analysis":   {"digitizable": 0.95, "verifiable": 0.90, "creative": 0.30},
        "writing":         {"digitizable": 1.00, "verifiable": 0.60, "creative": 0.50},
        "data_collection": {"digitizable": 0.25, "verifiable": 0.50, "creative": 0.40},
        "hypothesis":      {"digitizable": 0.50, "verifiable": 0.15, "creative": 0.95},
    }
    profile = dimensions.get(task)
    if not profile:
        return {}
    score = (w_digi * profile["digitizable"]
             + w_veri * profile["verifiable"]
             + w_crea * (1 - profile["creative"]))
    # Bidirectional effect of verifiability: apply an additional penalty when falsification requires long-term experiments (very low verifiability)
    if profile["verifiable"] < 0.3:
        score -= 0.15
    score = round(max(score, 0.0), 3)
    # Fallback strategy: high scores only prioritize automatic draft handling; outputs still require human review
    if score >= high_threshold:
        action = "Prioritize automatic draft handling, but outputs still require human review"
    elif score >= low_threshold:
        action = "AI-assisted, human-led"
    else:
        action = "Keep humans in charge; AI provides prompts only"
    return {"task": task, "ai_suitability": score, "action": action}

This code converts qualitative judgment into computable scoring logic: digitizability and verifiability serve as high-weight positive items, while creative requirement serves as an inverse constraint. In tools, such scoring can be used as the basis for routing strategies: high-scoring tasks can be prioritized for automatic draft handling, but outputs still require human review, while low-scoring tasks remain human-led.

5. Implications for Building Scientific AI Tools

1. Focus on validated high-fit scenarios and clarify boundaries. Priority should be given to deepening use cases such as data analysis, text writing, and literature review, rather than pursuing “coverage of the entire research workflow.” Data collection and hypothesis generation are better positioned as “assistive prompts” rather than “automatic decision-makers,” preserving the final judgment of humans.

2. Adopt a privacy architecture with strong compliance. Following the labeling and aggregation approach, research content should be stripped at the collection stage, with only classification metadata retained. Externally, only statistical views should be presented. Sensitive results such as unpublished data should not enter model training or storage pipelines.

3. Design for verifiability. For data-analysis outputs, built-in validation mechanisms such as recomputation, confidence intervals, and anomaly detection should be included so that conclusions are traceable and verifiable, reducing the risk of misuse.

4. Complete approval and de-identification before upload. Before users upload unpublished data, subject data, or data restricted by grant agreements, institutional approval and de-identification must be completed. Users must also know that AI outputs may fabricate references and statistics, so verification against original data and code is required.

5. Configure capabilities differently by discipline. Because task structures vary greatly across disciplines, model selection can follow the table below:

Discipline Type	High-Fit Tasks	Recommended Interaction Model
Quantitative fields (computer science, AI/ML, engineering)	Data analysis, modeling, code implementation	Provide recomputable code, unit tests, and result validation
Experimental sciences (physics, chemistry, biological experiments)	Data processing, literature review for experimental plans	Emphasize instrument-data import and de-identification; humans lead experimental operations
Clinical research (medicine)	Literature review, statistical assistance, document drafting	Emphasize de-identification, ethics approval, and human review of results

Five self-check questions for task fit (can be used directly with the scoring function in Section 4):

1. Can the input be digitized?
2. Can the output be recomputed at low cost?
3. How costly is an error in the result?
4. Is the task constrained by ethics or equipment?
5. Is human review required?

Score each item as high, medium, or low, input the result into assess_ai_suitability to calculate the fit score, and then use the thresholds to decide whether to use automatic handling, assistance, or human-led operation.

6. FAQ

• What can AI replace in research, and what can it not replace?
  AI conversations account for a high share in tasks that are digitizable and verifiable, such as data analysis and writing, and can take on drafting and assistance roles. In data collection and original hypothesis generation, the share is clearly lower, and humans still need to take the lead.

• What should researchers pay attention to when uploading data to AI?
  Before uploading unpublished data or subject data, institutional approval and de-identification must be completed. Sensitive results should not enter model training or storage pipelines, and AI outputs need to be verified against original data.

• How can one determine whether a research task is suitable for AI?
  Use the five questions: whether input can be digitized, whether output can be recomputed at low cost, the cost of errors, ethical and equipment constraints, and whether review is required. Score them, then combine them with the fit score and thresholds to make a judgment.

7. Conclusion

This observation report, based on large-scale conversation samples, provides a quantitative perspective for understanding the position of AI in scientific research. Its core conclusion is pragmatic: in AI conversations, data analysis and writing account for the highest shares and represent high-frequency usage scenarios, while data collection and original hypothesis generation account for clearly lower shares. For developers, rather than chasing the narrative of an “all-capable research assistant,” it is better to follow the main line of “digitizable, verifiable, low-creativity” and build tools that are solid, compliant, and trustworthy. Once again, it should be noted that the above data comes from a single-platform sample, so the applicable scope of the conclusions is limited and should not be directly extrapolated to all AI tools or to the entire scientific research population.