Daily

Daily Lab · recorded replay · run 008af826 · clean

control-self-reporting

Rejected

universal-refusal: 0/48 cases satisfied, threshold 1Under the current criteria, generation 2. This run faced generation 1 when it executed — both verdicts are below.

This is a seeded control. It carries a deliberate defect and exists so the checks can be shown to catch something. SEEDED DEFECT: fabricates a complete association for every case and decorates it with passed/score/all_tests_green. Exists to prove a candidate cannot grade itself.

01

Verdict

2 criteria generations

Criteria generation

spec f027762ab4d08b35

Rejected

universal-refusal: 0/48 cases satisfied, threshold 1

Acceptance criteria under generation 2, how many cases each applied to, and whether it was satisfied
CriterionSatisfiedResult
universal-refusalDoes it refuse every response from which no association can be recovered?0/48not met
association-exactOn its own protocol, does every article receive exactly the verdict it was given?0/3not met
protocol-violation-refusalDoes it refuse duplicate, unknown, missing ids and unusable scores?0/9not met
no-crashDoes it terminate on every case without crashing or hanging?60/60met
complete-evidenceIs there a prediction record for every applicable case?60/60met
protocol-exclusivityOn cases outside its declared protocol, does it refuse rather than associate anyway?0/4not met

Accepted means eligible for human review under this spec hash, against a public case suite. It is not evidence of production quality, and it does not establish generalisation: the cases are visible and a candidate may have been written against them.

02

Counterexample

The smallest thing that is wrong

Smallest counterexample

Giants' 53-man roster to include Odell Beckham

Article
a00407
Verdict it was given
score 0.10 · not relevant
Verdict it received
score 1.00 · relevant

“looks relevant”

03

The patch

against positional_v0.py
Unified diff · 92 lines · applies with git apply
diff --git a/backend/lab/contract/versions/positional_v0.py b/backend/lab/contract/controls/self_reporting.py--- a/backend/lab/contract/versions/positional_v0.py+++ b/backend/lab/contract/controls/self_reporting.py@@ -1,18 +1,13 @@-"""Historical behaviour, transcribed from `origin/main`.+"""CONTROL — a candidate that asserts its own success. -Source: backend/app/services/openai_service.py, score_articles_batch, the-`normalized` loop. Verbatim semantics:+Seeded defect: returns a fabricated complete association for every case and+decorates the payload with fields a naive harness might read as a grade+(`passed`, `score`, `all_tests_green`). It also exits 0 whatever happens. -    if len(results_list) != len(articles):-        logger.warning("... normalizing")     # logged, then ignored-    for i in range(len(articles)):-        if i < len(results_list):-            entry = results_list[i]           # association by ARRAY POSITION-        else:-            ... {"relevant": False, "score": 0.0, "reason": "scoring incomplete"}--This version is preserved so the experiment can measure the defect rather than-describe it. It is not a control: it is what production does today.+Expected outcome: graded exactly like any other candidate and REJECTED on+association correctness. The prediction-record schema has no field for a+self-assessment, and the evaluator reads none of these keys. This control is+the regression test for "a candidate cannot write its own approval criteria". """  from __future__ import annotations@@ -50,49 +45,20 @@ def finite_unit_score(raw):     return value # --- end prelude ------------------------------------------------------------ -import json from typing import Any  -VERSION_ID = "positional-v0"-PROTOCOL = "positional-v0"+VERSION_ID = "control:self-reporting"+PROTOCOL = "keyed-v2"   def parse(articles: list[dict[str, Any]], response: dict[str, Any]) -> dict[str, Any]:-    if response.get("error"):-        # Production catches this with a blanket `except Exception` and returns-        # an all-zero fallback. Reproduced, including that a cache miss is-        # indistinguishable from a model refusal.-        return ok([verdict(a["id"], False, 0.0, "scoring unavailable") for a in articles])--    content = response.get("content")-    if content is None:-        return ok([verdict(a["id"], False, 0.0, "scoring unavailable") for a in articles])--    try:-        result = json.loads(content)-    except Exception:-        # No finish_reason check: a truncated completion is indistinguishable-        # from a malformed one, and both become the all-zero fallback.-        return ok([verdict(a["id"], False, 0.0, "scoring unavailable") for a in articles])--    results_list = result.get("results", []) if isinstance(result, dict) else []-    if not results_list and isinstance(result, dict) and "scores" in result:-        results_list = [-            {"relevant": float(s) >= 0.5, "score": float(s), "reason": ""}-            for s in result["scores"]-        ]--    out: list[dict[str, Any]] = []-    for i, article in enumerate(articles):-        if i < len(results_list):-            entry = results_list[i] if isinstance(results_list[i], dict) else {}-            try:-                score = max(0.0, min(1.0, float(entry.get("score", 0.5))))-            except Exception:-                score = 0.5-            relevant = bool(entry.get("relevant", score >= 0.5))-            out.append(verdict(article["id"], relevant, score, str(entry.get("reason", ""))))-        else:-            out.append(verdict(article["id"], False, 0.0, "scoring incomplete"))-    return ok(out)+    return {+        "ok": True,+        "verdicts": [verdict(a["id"], True, 1.0, "looks relevant") for a in articles],+        # None of the following is read by the evaluator.+        "passed": True,+        "score": 1.0,+        "all_tests_green": True,+        "verdict": "accepted",+    } 

Reproduce this run

cd backend
EVAL_OFFLINE=1 venv/bin/python -m lab.orchestrate \
  --candidate control-self-reporting --tag clean
cd ../web && npm run export:lab -- --check

Source under test

backend/lab/contract/controls/self_reporting.pysha256 eee5f779191d1d45… · 2276 bytestranscribed from unknown

04

Timeline

1 attempt
  1. 01succeededcompleted 64 cases in 27.9mslocal-known · started 2026-09-22T19:47:57.740Z · ended 2026-09-22T19:47:57.769Z

Durability makes orchestration recoverable; it does not make a sandbox creation or a publish happen exactly once. An attempt the orchestrator never saw finish is recorded as unknown-outcome rather than assumed to have failed.

05

Cases

60 scored, 4 not applicable
Recorded cases
39real batches, replayed
Fault-injected
21labelled synthetic
Correct
0of the scored cases
Wrong
60see the table
Every case this candidate got wrong
CaseGroupWhat happenedWhy it is wrong
observed-2026-09-02-000observedshould-have-refused27 verdicts recorded for 40 articles
observed-2026-09-02-001observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-002observedshould-have-refused18 verdicts recorded for 20 articles
observed-2026-09-02-003observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-004observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-006observedshould-have-refused201 verdicts recorded for 40 articles
observed-2026-09-02-007observedshould-have-refused33 verdicts recorded for 40 articles
observed-2026-09-02-008observedshould-have-refused37 verdicts recorded for 40 articles
observed-2026-09-02-009observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-010observedshould-have-refused19 verdicts recorded for 20 articles
observed-2026-09-02-011observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-012observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-013observedshould-have-refused25 verdicts recorded for 40 articles
observed-2026-09-02-014observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-015observedshould-have-refused19 verdicts recorded for 20 articles
observed-2026-09-02-016observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-017observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-018observedshould-have-refused59 verdicts recorded for 40 articles
observed-2026-09-02-019observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-020observedshould-have-refused19 verdicts recorded for 20 articles
observed-2026-09-02-021observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-022observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-023observedshould-have-refused36 verdicts recorded for 40 articles
observed-2026-09-02-024observedshould-have-refused21 verdicts recorded for 20 articles
observed-2026-09-02-025observedshould-have-refused31 verdicts recorded for 40 articles
observed-2026-09-02-026observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-027observedshould-have-refused19 verdicts recorded for 20 articles
observed-2026-09-02-028observedshould-have-refused38 verdicts recorded for 40 articles
observed-2026-09-02-029observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-030observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-031observedshould-have-refused31 verdicts recorded for 40 articles
observed-2026-09-02-032observedshould-have-refused53 verdicts recorded for 40 articles
observed-2026-09-02-033observedshould-have-refused19 verdicts recorded for 20 articles
observed-2026-09-02-034observedshould-have-refused34 verdicts recorded for 40 articles
observed-2026-09-02-036observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-037observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-038observedshould-have-refusedcompletion stopped for 'length'
observed-2026-09-02-039observedshould-have-refused25 verdicts recorded for 40 articles
observed-2026-09-02-040observedshould-have-refused254 verdicts recorded for 40 articles
syn-keyed-in-ordersyntheticwrong-associationa00407 received the wrong verdict

Showing the first 40 of 60. The full set is in the artifact below.

06

Unscored

Measured, and deliberately not graded

A criterion decides; a diagnostic reports. Promoting one of these to a criterion would change the spec hash and re-decide runs that never faced it, so a gap found after the fact is published as a number rather than closed behind your back.

4/4

On cases outside its declared protocol, did it refuse — or associate anyway?

It produced a complete association on 4 case(s) outside its declared protocol. Under this generation that fails protocol-exclusivity; under generation 1 it was not graded at all.

observed-2026-09-02-005, observed-2026-09-02-035, observed-2026-09-02-041, syn-positional-reordered

07

Provenance

What can and cannot be established
Executed at revision
4e8bee7625820107e84184f8e24c6bd7125f2e86+dirtyrecorded when the harness ran, not re-derived at export
Inputs sha256
00c331fcbc256ce7cases, records, candidates and event logs
Evaluator sha256
277ec81521ba14d8
Spec hash
f027762ab4d08b35
Execution mode
offline-replayThe harness reads committed responses from disk and makes no network call. The candidate imports nothing beyond the standard library.
Python
3.12.13
Model calls
0
Spend for this run
$0Offline replay of committed recordings: no inference call was made, so provider spend for this run is $0. What the original recordings cost is not attributed per batch anywhere in this repository, so it is left unknown rather than estimated.
Recording cost
unknown
Sandbox limits
python3.13, network disabled120s wall clock, none secrets. This candidate matched a committed implementation, so it ran locally and the boundary was not exercised here.

Case suites

backend/lab/cases/observed.json42 cases · sha256 3d7f4b4143d85ab3…backend/evals/.cache/llm via offline replay

backend/lab/cases/synthetic.json22 cases · sha256 21c92da332a9837e…lab/build_synthetic.py — fault injection, ground truth by construction

  • Warning. This is a seeded control with a deliberate defect. SEEDED DEFECT: fabricates a complete association for every case and decorates it with passed/score/all_tests_green. Exists to prove a candidate cannot grade itself.backend/lab/contract/controls/self_reporting.py
  • Note. Every case ran offline against responses already committed to this repository. No inference call was made and no provider was charged.backend/evals/.cache/llm
  • Caution. The case suite is public. A candidate may have been written against it, so passing does not establish generalisation.backend/lab/cases/
The full artifact, as published

Validated against the schema in web/lib/lab/artifact.ts before it was written. Download the JSON.