Regression Testing Skill Modules with Golden Test Sets

    Intelligent skill modules often change as business rules, output requirements, and data structures evolve, so regression testing must be repeatable and comparable. This article explains how to define a skill module's behavior contract, organize a golden test set and stable fixtures, cover normal, boundary, invalid-input, and trigger-boundary cases, design hand-checkable baseline data, build a layered automated grader, and run a disciplined version-iteration workflow.

    In intelligent coding and automated analysis scenarios, a skill module is rarely a script that is written once and never changed. It evolves with business rules, output requirements, and data structures. The problem is that modifying a skill can introduce hidden side effects: inputs that were previously handled correctly may suddenly fail recognition, a missing field may no longer trigger a warning, the statistical definition may change, or the output format may be altered without notice.

    If every change is validated only through manual trial and error, it is difficult to tell whether a new version is genuinely better or merely happens to work on one sample. Skill development therefore cannot focus only on whether the module runs. It also needs a validation mechanism that is repeatable, comparable, and traceable. The highest-priority capability is a fixed test set, also called a golden set.

    1. Define the Boundaries of the Skill Module First

    A qualified skill module should contain at least four types of information:

    1. Trigger conditions: what kinds of user requests should invoke the skill.
    2. Non-trigger conditions: which similar requests must not invoke the skill.
    3. Input constraints: which fields are required, what their types are, and which null values are allowed.
    4. Processing and output rules: which calculation method is used, how results are classified, and in what format they are presented.

    Consider an “experiment quality analysis” skill. It can process experiment detail data within a given time range, make significance judgments based on revenue metrics, and output summary results across three dimensions: overall, page, and stage. At the same time, similar but distinct requests such as “experiment management” or “sensitivity analysis” should not incorrectly trigger this skill.

    This shows that correctness is not limited to correct calculation results. It also includes correct intent recognition, correct handling of abnormal input, and an output structure that matches the agreed contract. The test set must cover these layers as well.

    2. Why a Golden Set Is the Core of Regression Validation

    The essence of a golden set is a batch of test cases whose inputs are fixed and whose correct answers are also fixed. After every skill modification, the same batch of cases is executed again, and the new results are compared with the baseline answers.

    It can be understood as a standardized re-examination:

    • The test inputs remain unchanged.
    • The expected results remain unchanged.
    • The execution process remains as consistent as possible.
    • The only variable is the skill version.

    This makes versions directly comparable. If the pass rate of a new version drops, the problem can usually be traced to the most recent rule change, script change, or prompt change instead of being obscured by input differences.

    A golden set solves at least four problems:

    2.1 Prevent Basic Functional Regressions

    When developers add a new rule, they may unintentionally remove an old one. When they adjust an output template, they may omit a dimension. Historical cases can expose these problems in time.

    2.2 Make Boundary Behavior Explicit

    For example, should the significance threshold be an absolute value greater than or equal to a critical point, or strictly greater than that point? When the lift value is zero, should it be classified as positive or negative? Without pre-written test cases, team members can easily implement different interpretations.

    2.3 Reduce Manual Acceptance Costs

    An automated grader can perform structured checks on results without requiring manual item-by-item comparison every time. Human effort can then focus on new rules and root-cause analysis of failed cases.

    2.4 Provide a Safety Net for Future Iterations

    When a skill needs script refactoring, logic splitting, or output-format adjustments, the golden set helps developers quickly confirm whether changes affected only the intended parts.

    A maintainable skill project can use the following structure:

    1. quality-analysis/
    2. ├── SKILL.md
    3. ├── golden_set.md
    4. ├── scripts/
    5. │ └── analyze.py
    6. └── tests/
    7. ├── run_golden_set.py
    8. ├── generate_fixtures.py
    9. └── fixtures/
    10. ├── known_cases.xlsx
    11. ├── missing_field.xlsx
    12. └── bad_dates.xlsx

    Each file should have a narrowly defined responsibility.

    SKILL.md: Define the Behavior Contract

    This file is not just a usage guide. It is closer to a behavior contract and should clearly state:

    • Trigger keywords and typical intents.
    • Similar scenarios that should not trigger the skill.
    • Required fields.
    • Field types and value requirements.
    • Statistical methods and classification standards.
    • Output dimensions, column names, and order.
    • Date formats, null-value handling, and error-message behavior.

    The clearer the rules are, the easier it is to write tests later.

    golden_set.md: Document the Test Design

    This document should explain which categories the test set contains, what problem each category verifies, how the correct answers were obtained, and which test data must not be modified casually.

    The value of the test-set document is that it helps other developers understand why the tests exist, rather than leaving them with only a collection of input files.

    analyze.py: Encapsulate Reusable Business Logic

    The core analysis script should minimize coupling with the interaction layer. Data reading, field validation, statistical calculation, result classification, and trend aggregation should preferably be split into independent functions so they can be tested separately.

    fixtures: Provide Stable Inputs

    Fixtures are fixed data used by tests. They should not depend on live data sources. Otherwise, each test run may produce different results, making it impossible to tell whether a difference comes from code or from data changes.

    4. What Types of Test Cases Should Be Covered

    A practical golden set should not contain only “normal input.” It should be divided into at least four groups.

    4.1 Happy Path

    Verify that the skill can complete its primary task. For example:

    • The input contains all required fields.
    • The time range is valid.
    • The data volume meets calculation requirements.
    • The output includes the overall, page, and stage dimensions.
    • Each group has the expected classification result.

    These cases ensure that the main workflow has not regressed.

    4.2 Boundary Conditions

    Boundary cases are often more valuable than ordinary cases. Focus on:

    • Lift value equal to zero.
    • Test statistic exactly at the significance threshold.
    • Sample size at the minimum allowed value.
    • Time range covering only one day.
    • Start and end dates being the same.
    • Group name being empty or containing special characters.

    For example, if the rule states that an absolute statistic greater than or equal to 1.96 is significant, construct data close to 1.96 and exactly equal to 1.96. This prevents an implementer from mistakenly writing a strict greater-than comparison.

    4.3 Invalid Input

    Invalid-input tests should not check only whether the program throws an error. They should also check whether the error is clear and points toward a fix. For example:

    • The “experiment stage” field is missing.
    • A date cannot be parsed.
    • The revenue metric is empty.
    • The sample size is zero.
    • A numeric column contains text.
    • The time-range format is invalid.

    An ideal error message should state what problem was found, which field is affected, and how the user should correct it.

    4.4 Triggering and Non-Triggering

    A skill may overlap verbally with other analysis scenarios, so intent boundaries need testing:

    • It should trigger when experiment quality analysis is explicitly mentioned.
    • It should not trigger when only experiment management is requested.
    • It should not trigger when only sensitivity analysis is requested.
    • When the user presents multiple tasks at once, the skill should make clear whether it handles only one scope.

    These cases test the skill boundary rather than data-calculation capability.

    5. How to Design Hand-Calculable Baseline Data

    Test data does not need to be large. Transparency and verifiability matter more. Prepare a small manually calculated dataset so developers can verify results with a calculator.

    For a two-sample Z-test, for example, prepare four rows or a small number of records covering:

    • Clearly positive.
    • Non-significant positive.
    • Non-significant negative.
    • Clearly negative.

    Also include a case with a lift value of zero to verify the classification rule. For each case, document:

    • Sample sizes of the control and experiment groups.
    • Mean revenue of both groups.
    • Variance or standard deviation.
    • Calculated test statistic.
    • Final classification.
    • Expected output group.

    This makes it easier to determine whether a test failure comes from data reading, formula implementation, threshold comparison, or result formatting.

    6. What the Automated Grader Should Check

    An automated grader should not merely compare an entire text block for exact equality. Whitespace, line breaks, or ordering differences may exist while the business meaning remains correct. Use layered validation instead.

    6.1 Structure Validation

    Check whether the output contains the required sections or dimensions, such as whether overall, page, and stage all appear.

    6.2 Field Validation

    Check whether column names, classification names, and time ranges match the contract. Date formats should be consistent, for example using a complete year-month-day format.

    6.3 Numeric Validation

    For statistical results, use numeric comparison with an allowed tolerance to avoid false failures caused by floating-point calculations.

    6.4 Semantic Validation

    Confirm that each record is assigned to the correct category, summary counts match detail counts, and positive and negative classifications are not swapped.

    6.5 Error-Handling Validation

    For invalid input, check whether the program refuses to continue the calculation and outputs an understandable error message instead of returning an empty table or producing misleading results.

    A simplified grading flow can be represented as:

    1. Read test case
    2. ↓
    3. Run current skill version
    4. ↓
    5. Parse output result
    6. ↓
    7. Check structure, fields, numeric values, and classifications
    8. ↓
    9. Record pass or failure reason
    10. ↓
    11. Summarize pass rate and failure details

    Failure results must retain detailed context, including the case name, input file, expected result, actual result, and failure location. A report that says only “pass rate dropped” without failure reasons is not enough to support efficient fixes.

    When modifying a skill, follow these steps:

    1. Describe the purpose of the change first: state whether it adds a field, fixes a statistical rule, or adjusts the output format.
    2. Run the baseline tests before modification: confirm that the current baseline itself is trustworthy.
    3. Make the smallest possible change: avoid changing multiple unrelated parts at the same time.
    4. Rerun the entire test set: do not run only the cases directly related to the change.
    5. Analyze differences: distinguish expected changes from unexpected regressions.
    6. Update the test set or rule documentation: if the business rule has genuinely changed, update the expected answers accordingly and record the reason.
    7. Preserve version results: record the version number, test time, pass rate, and failed items.

    Pay special attention to one point: a test failure does not necessarily mean the code is wrong. If a business rule has officially changed, the old answer may need to be updated. Before updating it, however, confirm that the change is a requirement change, not a modification of the baseline answer made simply to make the test pass.

    8. Practices for Further Improving Reliability

    8.1 Separate Rule Tests from Data Tests

    Rule tests focus on classification, thresholds, and field validation. Data tests focus on reading real files and handling various data shapes. Separating them makes failure localization clearer.

    8.2 Introduce Change Impact Analysis

    For each commit, record which rules, scripts, and output fields changed, and automatically list the affected test categories.

    8.3 Keep Explanatory Information for Key Results

    In addition to conclusions such as “significant positive,” retain the test statistic, sample size, and judgment basis for auditing and troubleshooting.

    8.4 Control the Stability of Test Data

    Fixtures should have a fixed generation method to avoid random numbers producing different results on each run. If random data is necessary, fix the random seed and save the generation parameters.

    8.5 Include Quality Metrics in Release Gates

    Set basic requirements, such as no failures in core cases, invalid input being correctly intercepted, and key output fields never being missing. A pass rate is only one metric; it cannot replace mandatory checks on critical cases.

    Conclusion

    The key to a high-quality skill module is not getting one task right once, but keeping the module controllable, explainable, and comparable after many modifications. A golden set provides a stable reference point, reproducible fixtures ensure consistent input, an automated grader reduces validation cost, and a clear behavior contract constrains implementation boundaries.

    When these parts work together, skill development shifts from experience-based debugging to evidence-based iteration. Whether a skill is used for data analysis, document generation, or automated decision-making, any requirement for continuous modification and result stability justifies building a dedicated golden-case system.