Golden Set Regression Testing for AI Coding Skills: A Practical Guide
This article explains how to use a Golden Set to run regression tests on AI coding skills. It covers why a fixed, answer-known test set turns skill changes into a controllable variable, how to layer cases by risk (positive, boundary, exception, and negative cases), a recommended version-controlled directory structure, and how to build an automated grader that computes a pass rate. It also addresses fixture management, iteration discipline, common pitfalls, and a step-by-step checklist for getting started.
In AI coding environments, a skill is typically composed of prompts, rule constraints, scripts, and output templates. Unlike a traditional function, it has no clear signature, yet it changes just as often as requirements evolve: adding trigger conditions, adjusting field validation, refining salience rules, or tweaking the output format. Every small change can cause an existing capability to regress. When that happens, manual spot-checks alone make it hard to judge whether “the new version is more stable than the old one.”
The Golden Set is the engineering practice designed to solve this problem. Its core idea is simple: prepare a fixed set of test cases with known answers, rerun it after every skill change, and measure different versions with the same yardstick. The case set stays constant, the expected answers stay constant, and the only variable is the skill version. Results across versions become directly comparable, and fluctuations in the score can be attributed more accurately to the part that was changed.
1. The Value of a Golden Set: Turning Changes into a Controllable Variable
Modifying a skill is a routine part of real work. Whether you add a new output field or adjust a salience threshold, the change can affect the final result. Without a fixed regression set, teams can usually only rely on two forms of validation: manually opening a few samples to inspect the output, or depending on production feedback. The first offers limited coverage; the second is relatively costly.
The first value of a Golden Set is controlling variables. It requires fixed inputs, fixed expected outputs, and a fixed execution method, so that each regression run feels like retaking the same exam. A rise or drop in the pass rate no longer mixes in data differences or human-judgment differences, and can be attributed mainly to the change in the skill version.
The second value is comparability. Running the same case set on v1, v2, and v3 produces a set of pass rates, a list of failing cases, and a diff report. Instead of arguing over whether things “feel about the same,” the team can directly see which cases flipped from pass to fail and which boundary conditions are still uncovered.
The third value is accumulation. Every time a real problem is fixed, the corresponding scenario can be captured as a new Golden Case. Over time, the test set becomes more than an acceptance tool; it also becomes a living document of the skill’s expected behavior.
2. What Cases a Golden Set Should Contain
A usable Golden Set should not cover only “normal inputs.” A more sound approach is to layer cases by risk, with at least four categories:
Positive cases: guard against basic functional regressions
Pick representative tasks and specify clear inputs and expected outputs. For example, running experiment-quality analysis over a given date range, where the expected output includes three dimensions—overall, page, and step—each group containing four experiment types, plus a summary row. Positive cases confirm the core path has not been broken.Boundary cases: guard against off-by-one rule errors
These include a lift of 0, a sample size that just reaches the threshold, date ranges at the start or end of a month, and significance absolute values close to 1.96. Boundary cases are often the best at exposing off-by-one errors or formatting issues in rule implementations.Exception cases: guard against degraded error-input handling
For example, a missing key field, an empty ARPU, a sample size of 0, or an illegal date. The expected result is not necessarily a successful output; it can also be a clear error message, field-validation feedback, or a fallback strategy. The key is to lock down “how it should fail” as well.Negative and trigger-boundary cases: guard against false triggers and missed triggers
A skill usually has a defined scope. For scenarios it should not handle, such as “experiment management” or “sensitivity analysis,” prepare negative cases to verify the skill does not overstep. For synonymous phrasings, verify that the skill still triggers correctly. Trigger rules are also part of the skill’s behavior.
The expected answers in a case must be specific enough to be asserted automatically. For example: which dimensions the output must contain, whether the time format omits zero padding, and whether significance is based on a two-sample Z-test with |z| ≥ 1.96. Vague descriptions like “the result looks roughly correct” do not belong in a Golden Set.
3. A Recommended Directory Structure
A Golden Set should not be scattered test data; it should be version-controlled alongside the skill source code. A common structure looks like this:
ab-quality-analysis/├── SKILL.md # Trigger/no-trigger rules, field validation, significance rules, output template├── golden_set.md # Case list, usage discipline, fixture notes├── scripts/│ └── analyze.py # Core analysis script└── tests/├── run_golden_set.py # Automated grader, computes pass rate in one run├── generate_fixtures.py # Fixture generator, ensures reproducible data└── fixtures/├── known_cases.csv # Hand-crafted data, Z-test results verifiable by hand├── missing_field.csv# Missing experiment step field└── bad_dates.csv # Empty ARPU, zero sample size, invalid dates
Here, SKILL.md defines the skill contract: when to trigger, when not to trigger, how fields are validated, how significance is determined, and what the output template looks like. golden_set.md records the testing methodology: which cases exist, what each case verifies, how fixtures are generated, and what the run discipline is. run_golden_set.py is the automated grader that executes cases and computes the pass rate. The fixtures directory holds baseline inputs with known answers, ensuring consistent results across different machines and different times.
The point of this structure is separation of responsibilities: skill rules, test methodology, execution scripts, and test data each have a clear place. When you add new cases later, you don’t need to cram logic into the prompt; instead, you modify the test assets and add assertions.
4. Breaking Down Cases with a Quality-Analysis Skill
Suppose there is a skill for experiment-quality analysis. It classifies significance over experiment-detail data for a given date range and outputs categories such as significant-positive, non-significant-positive, non-significant-negative, and significant-negative, along with summaries across three dimensions: overall, experiment page, and experiment step.
A positive case could be designed like this:
- Input: run experiment-quality analysis for 2026/10/1–2026/10/31.
- Expected output:
- The output contains three dimensions: overall, page, and step;
- Each group contains all four experiment types plus a summary row;
- The date-range format is
2026/10/1-2026/10/31, with no zero padding on the month or day, and includes the year; - Significance is based on a two-sample Z-test of ARPU, with |z| ≥ 1.96 judged as significant;
- For four rows of hand-constructed data, the classification matches the hand-calculated result.
This single case covers functionality, formatting, and the algorithm rule at once. Its purpose is not complexity, but to quickly confirm after each change that “the core behavior has not changed.” If a change adjusts the output template, the time-format assertion will fail; if it changes the significance threshold, hand-verifiable boundary data will expose the problem.
Exception cases can use fixtures with a missing experiment-step field, an empty ARPU, a sample size of 0, or illegal dates. For each exception input, the expected behavior must be explicit: is it an error, an ignore, a default value, or a validation message? Only when the expected behavior is written clearly can the automated grader judge reliably.
5. The Automated Grader: Making the Pass Rate Computable
The job of an automated grader is not to reimplement the business logic, but to execute the skill, read the result, check it item by item against the assertions, and produce a readable report. A simplified pseudocode example:
def run_case(case):actual = invoke_skill(case.input)checks = []checks.append(assert_contains(actual, "overall"))checks.append(assert_contains(actual, "page"))checks.append(assert_contains(actual, "step"))checks.append(assert_regex(actual, r"2026/10/1-2026/10/31"))checks.append(assert_equals(actual.classification, case.expected))return all(checks)def main():cases = load_cases("golden_set.md")passed = sum(run_case(c) for c in cases)print(f"pass rate: {passed}/{len(cases)}")
In a real implementation, assertions should be split as finely as possible: structural assertions, field assertions, format assertions, and numeric assertions should be recorded separately. That way, a failure tells you not only that “the case failed,” but also whether it was a missing dimension, a wrong time format, or a mismatched significance classification. The report should ideally retain the failing case name, a summary of the actual output, and the expected diff, so issues are easy to locate.
The pass rate itself is not the goal, but a metric for version comparison. It is recommended to save the baseline result on every run—for example, the current version, run time, number of passing cases, and the list of failing cases. After the next change, compare against the baseline and focus on two categories: “newly failing” and “persistently failing.”
6. Fixture Management: Fixed, Reproducible, Hand-Verifiable
The credibility of a Golden Set depends heavily on its fixtures. Fixtures should meet three requirements:
First, fixed and reproducible. Test data should not depend on live APIs, random numbers, or the current date. When necessary, use a generation script, but the script itself should also be version-controlled, so the same version produces the same data.
Second, verifiable answers. Key cases should preferably use small-scale hand-crafted data, so that the Z-test results can be checked by hand. That way, when a test fails, the team can tell whether the skill implementation is wrong or the expected answer is wrong.
Third, representative risk coverage. Fixtures aim not for quantity but for hitting key rules: empty values, zero samples, illegal dates, boundary lifts, missing fields, cross-month date ranges, and so on. Each fixture should be bound to a clear case and expected behavior.
7. Iteration Discipline and Common Pitfalls
To keep a Golden Set effective over time, a few disciplines are needed: run the full Golden Set after every skill change; add or adjust cases whenever a new rule is introduced; keep fixed cases as regression assets; and require review for any change to the Golden Set—never modify expected answers just to make a test pass.
Common pitfalls include: testing only the happy path, so regressions in error handling go unnoticed; expected answers that are too vague, degrading automated grading into manual eyeballing; cases that balloon without layering, making runs slow and maintenance costly; fixtures that depend on the external environment, passing today and failing tomorrow; and treating the pass rate as the only quality metric while ignoring whether the output truly satisfies business semantics.
A more sound approach: positive cases guarantee core functionality, boundary cases guarantee rule precision, exception cases guarantee robustness, and negative cases guarantee the trigger scope. The pass rate is used for version comparison, the failing list is used to locate problems, and human review is used to judge whether the expected answers are still reasonable.
8. A Step-by-Step Checklist to Get Started
Getting a Golden Set off the ground can start with a small closed loop:
- Pick the single most core positive case for the skill, and write out the input and expected output clearly.
- Add 2 to 3 boundary and exception cases, ensuring the fixtures are reproducible.
- Write the automated grader to output a pass rate and a failing list.
- Save the current version’s result as the baseline.
- After each skill change, run the full case set and compare against the baseline.
- Capture real defects as new cases, and periodically review whether the cases are still valid.
The essence of a Golden Set is not adding testing burden, but moving skill iteration from “experience-driven” to “evidence-driven.” Only when every change can be rerun against the same set of known answers can a team evolve skills more reliably and more easily explain the reasons behind each change in the pass rate.
