Making Skills Stable Across Models and Frameworks: A Skill Virtual Machine Approach

1. Common Symptoms: The Same Skill Fails After Moving

If you have used coding assistants or general-purpose agent frameworks for any length of time, you have probably seen the following scenarios:

1. Changing the model breaks the skill. The same skill may run successfully on a flagship closed-source model, but fail when moved to an open-source model or a smaller local model. The model may repeatedly struggle with non-core decisions such as which file to read, how to fill in parameters, or what the output should look like.
2. Changing the framework distorts the skill. A skill that works reliably in one coding assistant may behave differently when migrated to another agent runtime framework. Tool names, call formats, and return structures may change, causing errors or broken output formats.
3. Changing the machine causes failure. Scripts that work on the skill author’s machine may fail on a colleague’s or customer’s machine with errors such as “command not found,” “module missing,” or “invalid path.” These issues often require several rounds of debugging to identify.

More subtly, these failures are often misdiagnosed as “the model is not smart enough.” As a result, teams keep switching to stronger models, increasing cost and latency without improving stability.

A recent university research study proposed a more fundamental view: skills are programs written in natural language, large language models are heterogeneous processing units, and the agent runtime framework is the execution environment. When a program fails after moving across heterogeneous processors, the problem is not a single model. The real issue is the mismatch among three layers: program, processor, and execution environment.

2. The Root Cause: Implicit Assumptions Hidden in Skill Documents

Strong models can often make a skill work because they infer missing information at runtime. For example:

• A document says “read the source file” but does not specify which directory the path is relative to, so the model has to try different locations.
• A document says “run the analysis script” but does not specify which interpreter to use or which dependencies are required, so the model has to discover missing pieces through errors.
• A document asks for an “output result list” but does not define the format, so the model produces arbitrary output that downstream programs cannot parse.

This self-rescue behavior is not free. Every additional probe, retry, or failed call consumes context, tool calls, and time. Weaker or local models do not have enough reasoning slack to recover from these gaps, so they stall directly.

When successful and failed cases are compared, failures usually fall into four categories of implicit assumptions:

Assumption Category	Typical Manifestation	Affected Models
Semantic assumptions	Instructions require the model to infer meaning, with no defined input format or boundaries	All models, especially weaker models
Tool assumptions	Tool names, parameters, and return structures vary by framework, and error messages are not standardized	All models
Environment assumptions	Paths, executables, dependencies, and permission policies vary by machine	All models
Output assumptions	Result format and success/failure criteria are unclear	All models

3. Reframing: Treat Skills as Programs Running on a “Language Virtual Machine”

This problem is similar to the evolution of computer architecture. The same source code can run on different processor architectures because a stable instruction set is abstracted away from hardware differences, and compilers or runtimes hide those differences.

Skills face a similar cross-model, cross-framework, and cross-environment problem:

Computing Analogy	Equivalent in the Skills World	Main Source of Differences
Source code	Skill instructions and scripts written in natural language	Free expression, often oversimplified
Instruction set / intermediate representation	Standardized capability descriptions and operation steps	Tool interfaces are not unified
Compiler / adaptation layer	Converts standard descriptions into forms executable by specific models and frameworks	Models differ in instruction-following ability
Processor	Large language model	Differences in context window, reasoning, instruction following, and structured output
Operating system / hardware	Agent runtime framework and execution machine	Differences in tool protocols, file systems, dependencies, and permission policies

Based on this analogy, a “skill virtual machine” should provide at least five capabilities:

1. Load: Read the skill package and parse metadata, dependency declarations, and output contracts.
2. Preflight checks: Before any business step runs, verify model capabilities, tool availability, and environment dependencies.
3. Instruction translation: Convert model-agnostic skill descriptions into prompts and tool calls that the target model and framework can understand.
4. Constrained execution: Execute with least privilege and record steps, inputs, outputs, and errors throughout the process.
5. Result validation: Validate outputs against the declared format and content requirements, with limited feedback-and-correction cycles when validation fails.

The “virtual machine” here is a borrowed design concept. Its value is not to make skills more complex. Its value is to move assumptions out of the model’s head and into places that machines can check.

Experiments described in the research, covering multiple model types and multiple runtime frameworks, showed that adding explicit capability descriptions, environment checks, and failure recovery improved execution stability across models and frameworks and reduced the number of retries. This suggests that stability can be achieved at the skill-engineering layer rather than only by switching models.

4. Scenario-Specific Diagnosis and Optimization

4.1 Cross-Model: Do Not Rely on Reasoning Slack; Put Requirements into the Instruction Layer

• Use structured instructions. Rewrite skill instructions with a five-part format: goal, constraints, inputs, output format, failure handling. Each step should contain only one action.
• Use example-driven instructions. Provide one or two input-output examples so the model aligns to the expected format instead of guessing.
• Do not rely on implicit common sense. When a step references a file, also explain how to locate it, such as relative paths or matching rules. Do not simply say “source file.”
• Probe capabilities. Before execution, verify whether the model can reliably produce structured output such as JSON. If capability is limited, use smaller step-by-step instructions instead of a long, complex skill description.

Goal: Aggregate CSV files in the data/ directory into a monthly summary table
Constraints: Process only files with the .csv extension; group by the "date" column into months; do not modify original files
Input: data/*.csv, where the first row of each file is the header
Output: Write to report/monthly.json as an array, where each item looks like:
      {"month": "YYYY-MM", "total": number, "count": integer}
Failure handling: If a file is missing or a field is missing, output {"error": "reason", "file": "filename"} and stop. Do not guess values.

4.2 Cross-Framework: Isolate Interface Differences with a Capability Abstraction Layer

Different frameworks provide different tool sets, parameter names, and return structures. A robust approach is to declare only standard capabilities in the skill and let each framework’s adapter handle mapping.

{
  "capability": "read_file",
  "input": {"path": "data/input.csv", "encoding": "utf-8"},
  "output": {"content": "string", "truncated": "boolean"},
  "on_error": ["FILE_NOT_FOUND", "PERMISSION_DENIED", "TOO_LARGE"]
}

On one side, the adapter maps read_file to the corresponding framework tool, command-line option, or local script call. On the other side, it normalizes diverse framework errors into unified error codes.

This design brings several benefits:

• When adding or replacing a runtime framework, only the adapter needs to be implemented. The skill itself does not need to be rewritten.
• When the skill encounters an unknown error code, it can apply a unified policy, such as retrying once or reporting the reason to the user, instead of falling into an infinite loop.
• During migration, interface equivalence tests can be run: execute the same set of inputs across multiple frameworks and compare whether the results are consistent.

4.3 Cross-Environment: Run Preflight Checks Before Execution; Avoid Blind Runs

“Works on the author’s machine” is one of the most common delivery traps. A skill package should contain the following information:

• Dependency manifest: Interpreter version, third-party libraries and version ranges, and required system commands.
• Preflight checks: Automatically probe the environment at startup and provide an actionable missing-item report.
• Path conventions: Use the skill package directory as the root and use relative paths. For system paths such as user directories or temporary directories, obtain them through a unified interface instead of hard-coding absolute paths.
• Permission boundaries: Define which directories can be read, which can be written to, whether scripts can be executed, and whether network access is allowed.
• Fallback strategy: If script dependencies are missing, fall back to “plain text steps plus standard tools” instead of exiting with an error.

#!/usr/bin/env bash
# preflight.sh: environment checks before skill execution
set -euo pipefail
root="$(cd "$(dirname "$0")" && pwd)"   # Use the skill package directory as root; avoid hard-coded absolute paths
fail=0
need_cmd python3 jq
for c in $need_cmd; do
  command -v "$c" >/dev/null 2>&1 || { echo "[Missing] Command not found: $c"; fail=1; }
done
if command -v python3 >/dev/null 2>&1; then
  python3 -c 'import sys; assert sys.version_info >= (3, 9), "Python 3.9+ is required"' 2>/dev/null \
    || { echo "[Missing] Python version is below 3.9"; fail=1; }
  python3 -c 'import pandas' 2>/dev/null || { echo "[Missing] pandas is not installed"; fail=1; }
fi
[[ -r "$root/data/input.csv" ]] || { echo "[Missing] Cannot read input file data/input.csv"; fail=1; }
(( fail == 0 )) || { echo "[Conclusion] Environment requirements are not met. Execution stopped. Fix the issues above and retry."; exit 1; }
echo "[Conclusion] Environment checks passed."

Moving preflight checks earlier has two benefits. First, the model does not need to discover missing items through repeated error messages, which saves multiple tool-call rounds. Second, the report is human-readable and actionable, so the user can fix the environment in one pass.

4.4 Cross-Scenario Output: Replace Free-Form Output with Contracts

Downstream consumers such as programs, reports, and other skills need stable structures. Therefore, a skill should explicitly declare its output contract:

{
  "format": "json",
  "schema": {
    "type": "object",
    "required": ["status", "data"],
    "properties": {
      "status": {"enum": ["ok", "partial", "failed"]},
      "data": {"type": "array"},
      "errors": {"type": "array"}
    }
  },
  "rules": ["Do not output fields outside the schema", "Do not use comments", "Do not include thousand separators in numbers"],
  "on_violation": "Return validation errors to the model for correction up to 2 times; if it still fails, return the original output with status failed"
}

Validation can be performed by a script or a JSON Schema validator, independent of the model. This provides at least three benefits:

• Consistent formats mean downstream systems do not need to support multiple output variants.
• Failures are deterministic, so invalid output can be detected promptly.
• Corrections are bounded, preventing endless retry loops.

5. Implementation Path: From Point Fixes to Systematic Governance

1. Inventory: Collect existing skills in the team, measure the proportion of failures caused by cross-model issues, cross-framework issues, and environment issues, and identify the skills with the highest potential benefit.
2. Add contracts: Add metadata, dependency manifests, input/output formats, and failure-handling rules to skills. Prioritize high-frequency skills.
3. Build adapters: Support at least two model families and two runtime frameworks, and verify that behavior remains consistent after migration.
4. Create an evaluation set: Prepare multiple typical inputs covering normal cases, boundary cases, and exceptional cases. Run automatic regression tests after each change.
5. Monitor and measure: Collect and continuously track the following metrics:

Metric	Meaning	Optimization Direction
First-pass success rate	Proportion of tasks completed successfully without human intervention	Continuously improve
Average retry rounds	Average number of errors and corrections before success	Approach one retry
Step-level success rate	Proportion of steps that succeed on the first execution	Used to locate weak points
Result consistency rate	Proportion of identical results across models and frameworks	Used to evaluate migration quality
Resource cost per task	Processing-unit calls and execution time consumed	Used to control cost

1. Version management: Put skill instructions, dependency manifests, and contracts under version control. Record the reason for each change and bind changes to evaluation-set results.

6. Engineering Recommendations: Security, Cost, and Operations

• Least privilege: Grant only the necessary read, write, and execution permissions when running a skill. High-risk operations such as network access should require explicit user confirmation.
• Trusted sources: Third-party skills may contain prompt-injection risks. For example, a malicious document may induce the model to perform unauthorized actions. Review skills before importing them.
• Explainable failures: Every failure should report which step failed, why it failed, and how to fix it, rather than producing a vague natural-language description.
• Resource isolation: Batch or long-running tasks can be executed in isolated environments, such as containers, to prevent interference between tasks.
• Cloud hosting: Skill packages and versions can be centrally managed in object storage. Execution traces can be retained in logging services for analysis, and metrics such as failure rate and retry rate can be continuously tracked through monitoring and alerting. For example, Baidu AI Cloud provides these as generic cloud service capabilities without relying on a specific vendor.
• Progressive splitting: Complex skills should not be written as one extremely long step. They can be split into sub-skills such as collect, process, validate, and output, making it easier to locate problems, test independently, and reuse components.

7. Conclusion

A skill that stops working after switching models usually outsources too many implicit assumptions to the model. Strong models can absorb these gaps, but at the cost of more resource consumption. Weaker and local models cannot absorb them, so the task fails directly.

Inspired by recent university research, treating skills as programs running on a “language virtual machine” and stabilizing their behavior through explicit contracts, capability abstraction, environment preflight checks, and result validation is a more sustainable solution.

For skill authors, this means upgrading the deliverable from “a prompt” to “a portable and verifiable skill package.” For platform teams, stability shifts from relying on model capability to measurable and regression-testable engineering metrics. For enterprise users, the same set of skills can be reused reliably across different models, runtime frameworks, and environments, significantly reducing trial-and-error costs and migration costs.