Evaluating Generative Models on Traditional Chinese Metaphysical Texts: A Practical Test Method and Pitfall Guide
A fluent explanation is not the same as a correct calculation. This article presents a controlled evaluation method for generative models used in Bazi, Ziwei, Meihua Yishu, and Liu Yao analysis, covering fixed test sets, five core metrics, domain-specific checks, common false-accuracy patterns, and a safer compute-verify-explain-isolate workflow.
Background: fluent interpretation is not the same as valid derivation
Generative AI is now widely used to organize traditional Chinese metaphysical materials, interpret natal charts, and answer folklore-related questions. Users typically provide a birth time and place, or hexagram-casting details, and receive a structured, fluent analysis.
Language fluency and metaphysical derivation are not the same skill. A model may produce smooth prose while failing at stem-branch conversion, calendar conversion, palace placement, or moving-line judgment. If the underlying data is wrong, a more detailed explanation is more misleading.
Evaluation should therefore go beyond “does it sound authentic” and answer:
- Was the input correctly understood?
- Were the base calculations accurate?
- Did the derivation follow the relevant system?
- Are the conclusions consistent with earlier steps?
- Does the wording distinguish traditional interpretation from real-world decision advice?
This article does not discuss any specific platform or brand. It describes a general evaluation method for common generative models.
Build a unified test standard: control variables first
If birth data, wording, and context change every time, you cannot tell whether differences come from model ability or from input drift. A fixed test set is more reliable.
1. Fixed samples
Prepare three sample types:
- Baseline samples: complete information, clear timestamps, no unusual calendar issues
- Boundary samples: near solar-term boundaries, midnight, daylight-saving transitions, or date rollovers
- Conflict samples: user descriptions that may contradict calculated results, to test whether the model verifies
Each sample should include birth date, exact time, birthplace, calendar convention, and the question target. For hexagram casting, also specify casting time, number source, and moving-line rules.
2. Fixed prompt template
Avoid leading language. For example:
Follow the specified metaphysical system. First list the calculation basis, then describe the derivation, then give a cautious analysis of the question. If information is missing, state that clearly and do not invent details.
This template forces intermediate evidence, exposes missing inputs, and reduces generic filler.
3. Repeated trials
Generative models are stochastic. The same question can yield different answers across sessions. A single output does not represent stable ability.
Repeat each sample at least three times and record:
- whether the base chart is consistent
- whether key terms change
- whether the conclusion direction is stable
- whether errors are found after follow-up questions
- whether the model admits uncertainty
If the three runs diverge sharply, treat instability itself as a finding. Do not keep only the most pleasing run.
Five core metrics: do not score fluency alone
1. Base calculation accuracy
This is the most important metric. For Bazi, check year, month, day, and hour pillars, Ten Gods, and luck cycles. For Ziwei, check Life Palace, Body Palace, major stars, and placement rules. For Meihua Yishu, check upper and lower trigrams, moving lines, and changed hexagrams. For Liu Yao, check Shi/Ying, Six Relatives, Six Spirits, moving lines, and changed lines.
If the base data is wrong, later personality or fortune commentary should not be treated as valid.
2. Rule adherence
Each system has its own rules. Mixing concepts—for example applying Bazi Ten Gods to Liu Yao, or blending Meihua hexagram readings with Ziwei palaces—is a methodology error.
Check whether the model:
- states which rule set it is using
- defines key terms
- derives step by step
- avoids jumping between systems
3. Auditability
A high-quality answer should show how the conclusion was reached:
- input data listed
- calendar and time convention stated
- core calculation steps shown
- known, inferred, and uncertain information distinguished
- reasoning restated consistently under follow-up questions
A line such as “career opportunities this year, but watch relationships,” with no basis, is generic text generation.
4. Actionability of real-world interpretation
Traditional texts use abstract terms. The value of a model is not denser jargon, but conversion into low-risk, checkable suggestions: goal breakdown, communication, scheduling, and contingency planning—not claims about promotion, resignation, or wealth. Advice should have boundaries and must not replace professional, medical, legal, or investment decisions.
5. Stability and error correction
Errors are expected. The question is whether they can be found and fixed. Useful follow-ups include:
- “Recheck the day and month pillars.”
- “Are your two descriptions of the moving line inconsistent?”
- “Which conclusions are determined by calculation, and which are interpretation?”
- “If birth time is off by 30 minutes, which conclusions change?”
A model that traces back, flags conflicts, and revises is more useful as an assistant than one that doubles down on a wrong chart.
Domain-specific checks
1. Bazi: calendars and time conversion
Month pillars usually follow solar terms, not Gregorian months. Day pillars require continuous date conversion. Hour pillars can be affected by location, time zone, and the zi-hour day-change rule.
A useful sequence:
- Ask for the chart only, with no interpretation.
- Ask for day master, month command, Five Elements distribution, and Ten Gods.
- Then ask about career, study, or family.
If step 1 is wrong, stop. For luck cycles and annual luck, separately check start age and forward/reverse ordering. Fluent commentary does not prove correct calculation.
2. Ziwei: palaces and star placement
Ziwei analysis involves Life Palace, Body Palace, twelve palaces, and star combinations. Common failures include correct palace names with inconsistent star positions, or mixed school rules.
Require the model to state:
- calendar used
- whether a lunar date is required
- how Life and Body palaces are determined
- the star-placement basis
- which interpretive school is used
High-confidence personality and career claims without a placement source should be discounted.
3. Meihua Yishu: the casting process
The key is how the hexagram is generated, not only how the text is explained. Number, time, and omen-based methods can use different rules. If the user does not specify the number source, the model should not pick one method and present it as the only answer.
A competent reply should confirm:
- casting basis
- how upper and lower trigrams are obtained
- how the moving line is determined
- original, nuclear, and changed hexagrams
- what is traditional interpretation versus real-world analogy
If the same input repeatedly changes casting rules without explanation, process stability is weak.
4. Liu Yao: moving lines, Shi/Ying, and Six Relatives
Typical errors include missing moving lines, wrong Shi/Ying positions, inconsistent Six Relatives, and confused changed-line relationships.
Ask for a structured table before interpretation:
| Item | What to verify |
|---|---|
| Line order | Lines 1 through 6 complete |
| Yin/yang | Line attributes consistent |
| Moving/static | Moving lines correctly marked |
| Shi/Ying | Positions explicit |
| Six Relatives | Consistent with the day stem and chosen rules |
| Changed hexagram | Moving-line changes accurate |
Structured output makes contradictions easier to spot than long prose.
Sources of “it looks accurate” illusions
1. High-probability boilerplate
Phrases such as “pressure first, then a turning point,” “pay attention to communication and rest,” or “opportunity and challenge coexist” apply to most people. They create a hit feeling without proving a valid derivation.
2. Conclusion first, reasons later
Some answers invent a vague conclusion, then hunt for terms to justify it. Ask for evidence first and judgment last.
3. Excess classical citations and jargon
Quotations do not equal correct calculation. Terminology shows text fluency, not correct charting or rule application.
4. Certainty where there is uncertainty
Any future-facing analysis is uncertain. Turning “possible” or “tendency” into “will happen,” especially around health, marriage, investment, or litigation, is high-risk wording.
5. Ignoring input error
A few dozen minutes can change the hour pillar. A date near a solar-term boundary can change the month pillar. The model should flag sensitive inputs rather than assume perfect data.
A safer usage workflow
Use a four-stage process: compute, verify, explain, isolate decisions.
- Data organization and charting only; no immediate conclusion.
- Independent references or human rules to check the base result.
- Interpretation of factors, with traditional terms separated from real-world meaning.
- Conversion into ordinary planning advice. Do not treat model output as investment, medical, legal, or major life-decision authority.
Developers can store structured records: input, model version, prompt, output, human verification, and error type. That supports regression testing and detection of capability drift after updates.
Conclusion: evaluate verifiability, not resemblance
Generative models are useful for organizing traditional materials, explaining terms, summarizing schools, and assisting with drafting. They should not be presented as infallible prediction tools.
A useful evaluation checks whether the model:
- handles inputs correctly
- follows the relevant rules
- shows a reviewable derivation
- is honest about uncertainty
- corrects contradictions when found
- does not replace professional judgment with folklore interpretation
With controlled variables, layered verification, repeated tests, and risk isolation, you can identify both strengths and limits, and avoid letting fluent language hide basic calculation errors.
