Golden Evidence Model¶

The Golden v1 model defines how this repository represents an experiment, the evidence it collects, and what that evidence is allowed to claim. Microsoft Learn documents how App Service behaves. This model exists to answer a different question: how does an engineer reproduce that behaviour, observe it, and know what the observation does and does not prove?

The chain¶

Experiment --collects--> Evidence --> Assertion --> Hypothesis

Each arrow is deliberately one-way. Collectors gather facts and never decide a cause. Assertions are declared before execution, so a run cannot choose its success criteria after seeing the result. The hypothesis verdict is derived from assertion outcomes, never asserted directly.

Vocabulary¶

Inferred may never be promoted to Observed. The evaluator raises rather than allowing it, because an inference that acquires the authority of an observation is indistinguishable from a measurement that was never taken.

Three rules that carry the model¶

Execution status is separate from hypothesis status¶

A run that did not finish has not tested anything, whatever its partial evidence happens to show. Folding the two axes together produces the most common failure in this repository's history: a hypothesis derived from the evidence verdict, which leaves CONTRADICTED unreachable and silently collapses four documented outcomes into two.

Absence is not refutation¶

Evidence that cannot decide an assertion yields INCONCLUSIVE. It never yields CONTRADICTED. A missing field, a truncated capture, and a log that has not arrived yet are all states of not knowing. Treating them as disproof manufactures findings that the evidence does not support.

INCONCLUSIVE is a normal, publishable result.

Raw evidence wins over any generated verdict¶

A run may carry a result.json. The evaluator never reads it as input. It recomputes the outcome from raw evidence, compares, and reports generated_verdict_rejected on disagreement. A verdict file that can decide its own outcome is not evidence.

Timestamps¶

Three distinct instants, never interchangeable:

validation.az_cli.last_tested and validation.bicep.last_tested are intended to represent the UTC calendar date on which the corresponding tutorial method was last executed. For multi-phase runs, last_tested uses the UTC calendar date on which the final required phase completed (successfully or with failure). They are not alternate names for captured_at, evaluated_at, or document_updated_at. Validation frontmatter uses a closed schema: only az_cli and bicep methods, the documented result vocabulary, valid non-future dates, and a consistent not_tested state are accepted. A new or changed pass entry must name an existing retained package through canonical run_id and evidence fields. This existence check does not prove that the retained package executed the named tutorial or method. Binding retained evidence to an actual tutorial execution remains open in issue #176.

The vocabulary migration from both stacked implementations, PR #186 and PR

187, was completed when they were reconciled into PR #234. The exact retired¶

to frozen compatibility mapping remains in the migration specification.

Re-running an offline evaluator over an older cohort must not restate that evidence as freshly captured. A field named for capture that actually holds evaluation time will silently republish months-old evidence as current on every re-run.

Run layout¶

evidence/<run-id>/
├── manifest.json     run identity, timestamps, declared assertions
├── baseline/         raw evidence, healthy state
├── fault/            raw evidence, fault applied
├── recovery/         raw evidence, after remediation
├── assertions/       per-assertion raw inputs
└── result.json       derived, regenerable, never an input

Run directories are immutable and never overwrite a previous run. A collector re-run creates a new run-id. CI compares existing files under evidence/runs/<run-id>/ and labs/*/golden/ with the pull-request base and rejects edits, renames, and deletions while allowing newly added run directories. Retirement is an explicit archive operation in a separately reviewed change; it is never an in-place rewrite or a silent deletion.

evidence/retained-digests.json anchors every retained file by SHA-256. Existing entries are append-only relative to the pull-request base, and digest checks run even when the comparison base is HEAD, so re-dating a package cannot be hidden by consistently rewriting its manifest, evidence, and result.

Evaluator¶

scripts/golden/evaluate_run.py is stdlib-only and re-derives every assertion outcome from raw evidence.

python3 scripts/golden/evaluate_run.py evidence/<run-id>

A refutation does not share exit 0 with a confirmation. Exit 0 reads as "the reproduction worked", and a disproof is a different result, not a successful one.

The evaluator does not name a root cause. It reports whether declared assertions held. For non-historical COMPLETE causal runs, provenance includes a source commit that resolves in this repository plus named subscription kind, region, runtime, and SKU fields. The explicit --skip-source-commit-resolution option is reserved for environments without Git object access and records source_commit_resolution: SKIPPED_EXPLICITLY in the result.

Schemas¶

Independent reproduction¶

A scenario can be classified as independently reproduced only when a second operator reaches the same verdict from the documentation alone and an external verifier confirms the operator identities.

The reviewer receives the repository, a target commit, the public documentation and a working environment. The reviewer does not receive hidden steps, undocumented workarounds, private notes, or any statement of the expected root cause. Telling a reviewer what to find converts the exercise into confirmation.

Two results are recorded separately, because they can diverge:

A reviewer who had to read source to fill a gap has shown that the code works and the documentation does not. The verdict may match while the documentation result is FAILED. Any deviation the reviewer had to invent counts the same way: a step absent from the documentation is a gap, whatever the outcome.

A replay is not independent if it reuses the original run identity or is performed by the original operator. It would agree by construction, because it is the same evidence.

Self-declared human strings do not prove identity. A record may cite a GitHub pull-request review or approval URL with the reviewer, original operator, and PR author logins, but offline checking returns ATTESTATION_UNVERIFIED unless a verifier hook resolves that reference and confirms all three logins are distinct. Existing agent-authored records remain NOT_INDEPENDENT; this repository does not claim a human reproduction from author-controlled metadata.

The CLI remains offline and returns ATTESTATION_UNVERIFIED; qualification is a maintainer decision outside this author-controlled record. The current retained run packages are explicitly historical in evidence/retained-digests.json; they are integrity-protected illustrations and cannot support a qualifying human reproduction record.

python3 scripts/golden/check_reproduction.py evidence/reproductions/<scenario>.json

Exit 0 requires both results to be REPRODUCED. An attempt that never happened is recorded as NOT_ATTEMPTED on both axes rather than omitted, so the gap is visible in the repository instead of inferred from absence.

Local Golden test dependencies¶

Create an isolated environment and install the same Python dependencies as CI:

python3 -m venv /tmp/azure-app-service-golden-venv
/tmp/azure-app-service-golden-venv/bin/python -m pip install jsonschema referencing requests pyyaml ruamel.yaml
/tmp/azure-app-service-golden-venv/bin/python -m unittest discover -s tests/golden -p 'test_*.py'

Predicate guidance¶

A gate whose weak path subsumes its strong path is unsound. Written as strong or fallback where fallback is one of strong's conjuncts, boolean absorption reduces the whole predicate to fallback alone, and every other condition becomes dead code while the gate keeps its reassuring name.

This is not hypothetical. It is the defect class that allowed a verdict file to pass a sub-gate whose raw data contradicted it.

Write the weak path so it cannot subsume the strong one, or place the shared invariant outside the choice:

# Unsound: reduces to `fallback` alone.
result = (a and b) or b

# Sound: the invariant applies to both paths.
result = (strong or fallback) and invariant

Threat model and residual limits¶

Offline gates verify internal consistency, source-bound recipes, retained raw-row derivations, and append-only package identity. They cannot prove that a pull request author did not forge an entirely new run before the package entered the repository. Comparison with the privately retained raw capture and the reviewer protocol are the control for that residual risk. A passing offline replay must not be described as independent proof that Azure execution occurred.

Tutorial frontmatter may refer to a retained package, but the offline gate checks only that the path is canonical and the directory exists. It does not enforce a tutorial/method binding and must not be presented as proof that the package is the tutorial's execution record.

source_commit is the runner-captured commit and proves only that the executed recipe was not committed after capture. Offline transformation is a separate fact: the derivation record identifies the later collector commit, its exact bytes, the honest derivation time, and the private raw phase hashes used as inputs. Neither timestamp substitutes for the other.

Historical evidence policy¶

A capture is evidence about the moment it was taken, not a standing fact. This policy says how long one stays usable and what it takes to renew it.

Two axes, deliberately separate¶

Freshness and provenance answer different questions, and a single field answering both answers neither. One field named trust once carried both vocabularies, and a classification was found holding values from each in the same place.

retention_status says how fresh an artifact is and whether it may still be relied on:

capture_provenance says how an observation was obtained:

Where a value came from does not say how stale it is, and how fresh an artifact is does not say whether anyone observed it directly.

A capture predating the contract it is cited under is trusted-historical, never trusted-current solely by re-reading. Reclassifying it upward requires re-capture, not re-reading.

Retention and re-verification¶

A capture stays authoritative until the platform behaviour it records changes. Because that is not observable from the capture, each Golden scenario states what would invalidate its evidence rather than asserting an expiry date. Re-verification means running the contract again and producing a new run directory; editing an old one is forbidden, since a run directory is the record of a moment.

Retained captures are never edited to match a new contract. When a contract changes so that an old capture no longer satisfies it, the capture is reclassified and the gap recorded in known_gaps, so the shortfall stays visible instead of being silently absorbed.

What a retained capture may not do¶

It may not satisfy an assertion its collector never attempted. The distinction matters most for discriminating assertions: an old capture that did not record the competing explanation cannot rule it out, and treating its silence as agreement converts missing data into support.

The run directory¶

A run is one directory. evaluate_run.py reads exactly two files from it:

result.json, when present, is recomputed and compared, never consumed. A collector may write additional files, and the evaluator ignores them: they are provenance for a reader, not inputs to a verdict.

A scenario may also carry its own runner with its own capture layout. The startup-failure lab writes run.json and per-phase directories through run.py, which evaluates against contract.json rather than against a Golden manifest. Those two layouts are bridged, not merged: a collector such as scripts/golden/collect_scenario_a.py reads the runner's directory and re-expresses its captures as the evidence.json this model describes. An independent reproduction reported this relationship as unexplained, which it was.

See Also¶

• Content Validation Status
• Validation Status

Sources¶

• Azure App Service documentation