ctipilot.ch

tools/check_brief.py — common FAILs and how to fix them

Phase 5.5 (daily) / Phase 4.5 (weekly) of the Claude Code routines runs python3 tools/check_brief.py as a non-negotiable gate before the verification sub-agent (Phase 5.7 daily / Phase 4.7 weekly) and again between every verification fix iteration. The script is read-only — it surfaces drift, you fix it. Below is the operator playbook for the common FAIL patterns the script emits.

The full check list (1–19) lives inline in prompts/daily-cti-brief.md § Phase 5.5; this doc is the remediation handbook.

How to fix common FAILs

FAIL Fix
cve-sync: missing from cves_seen.json: [...] Append the listed CVE entries under § Phase 5 / state/cves_seen.json (historical-context CVEs and deferred CVEs count too — anything with a CVE-… token in the brief).
footer-presence: items without v2 footer Re-Edit the affected H3 to append a — *Source: [Title](URL) · Tags: … · Region: …* line.
run-log-fields: ... missing keys Rewrite today's record in state/run_log.json to match the schema in § Phase 5 (sub_agents, fetch_failures, items_published, deep_dive, verification_iterations, verification_residual_count are all required).
run-log-subagents: sub-agent records incomplete Each of S1, S2, S3, S4 must have `{sources_attempted: [...], sources_used: [...], items_returned: N, returned: true false}. Empty arrays are fine on a stalled sub-agent (returned: false`).
sources-touched: no source has last_successful_fetch == today Update last_successful_fetch on every source you actually fetched today.
footer-taxonomy: unknown ... Either correct the footer or extend site/taxonomy.yaml in the same commit.
fetch-source-403: 403/429 on known-403 hosts not mitigated Re-run the affected URL via python3 tools/fetch_source.py … and update the source bookkeeping.
multi-cve-cvss: N CVEs but single CVSS Either confirm both CVEs share that CVSS (single value is fine) or write per-CVE: CVSS: 9.1 / 7.2 or CVSS: 9.1 (CVE-…), 7.2 (CVE-…).
blocked-source: ... cites https://nvd.nist.gov/vuln/detail/CVE-… Replace with the vendor PSIRT advisory or research blog. NVD/MITRE/cve.org per-CVE pages are blocked as Sources — they are derived. The build still surfaces them automatically as External References on every per-CVE page.
blocked-source: ... cites <publisher>/news/ (or any landing) Re-fetch and link the specific article URL (i.e. the article's own page with its own slug, not the section landing). Generic landings are not Sources.
source-urls: <url> returns 404 The URL is fabricated or moved. Re-do the primary-source pivot (WebSearch for the topic, fetch the result, swap the citation). If the original primary genuinely doesn't exist, drop the item.
anchor-resolution: N internal anchor link(s) do not resolve to an H2/H3/H4 slug (v2.52) Each broken anchor is reported with its visible link text and the attempted #slug. Either (a) the target heading was renamed in a prior iteration — find the current heading and recompute the slug using site/build.py's rule (lowercase, replace [^a-z0-9]+ with -, collapse runs, strip), or (b) the slug has a double hyphen (--) the slugify collapses to one. Update the [text](#slug) reference in § 6 Action Items. Never paper over by changing the heading to match a stale anchor — that breaks every previously-published external link. The check matches what the static site at ctipilot.ch actually generates, so a passing brief has working anchors.
verification-final-verdict-set: verification.final_verdict = 'pending' (v2.53) The verifier has not returned (or returned PENDING / unknown) and the commit-gate refuses to let the brief publish. Wait for the verifier to actually return — do not flip final_verdict to CLEAN manually to satisfy the gate. If the verifier hard-timed out past the 30-min cap, record final_verdict: "timeout" plus a § 7 / § 10 Verification Notes line explaining the gap; the gate accepts "timeout" because that's an honest terminal state. If a stop hook forced an early commit and the verifier is genuinely still in flight, finish the iteration (apply remediations, record the verdict, re-run tools/check_brief.py) before the next commit. The 2026-05-15 run's premature commit kept the dangerous Datadog inversion on main for several minutes; this check prevents that.
evidence-shape: N item(s) have an Evidence: field but no parseable quotes (v2.53) The footer carries Evidence: but no "quote" (Publisher) pairs were extracted. Check the punctuation: each quote must use straight double quotes ("..."); each attribution must be in parentheses immediately after the closing quote. Multiple quotes separated by ; (not ·, which the parser uses as the field-separator). Example shape: · Evidence: "Cisco Talos is tracking the active exploitation" (Cisco Talos); "CVSS v3.1 score of 10.0" (Rapid7) ·. Bad shape (no quotes parsed): · Evidence: see Talos and Rapid7 ·. If the field was added experimentally and you don't have quotes yet, just remove the field — items without Evidence pass silently in v2.53.

WARN-level signal (not blocking)

  • primary-source-quality WARNs — items whose first source is NVD or a CERT/NCSC. Re-pivot to the vendor advisory / research-lab post / vendor blog and put NVD or the CERT as Additional source: instead.
  • covered-items WARNs — covered_items.json drift relative to the brief. The next run rebuilds from the brief, so this is observability, not a hard block.
  • quantifier-evidence WARNs (v2.52) — Surfaces phrases like "first time", "the only", "five unpatched zero-days", "10 additional clusters" for verifier corroboration. Each WARN names the H3 item, the matching pattern category (absolute-claim, numeric-status, cluster-count, at-least-count), and a short prose snippet. The fix is not to remove the quantifier; the fix is to confirm the cited source contains the same quantifier. If the source supports it, keep it; if not, soften (e.g. "five" → "several" without an enumerated count) or drop the quantifier. Suppress legitimate cases — repeating a quantifier the cited source uses verbatim — by leaving the WARN unresolved at commit; the gate is non-blocking. The verifier reads the WARN list during its truth pass and will downgrade those items as well if the source backs them. Once Tier 1 evidence-binding ships, this WARN becomes a FAIL gated on the per-claim source quote.
  • tldr-body-drift WARNs (v2.52) — TL;DR bullets whose regional phrasing diverges from the body footer for the same CVE. v2.52 ships a Swiss-and-European-only phrase set: if the TL;DR mentions "Swiss", "Switzerland", "CH government", "EU/CH", or similar, the body footer's Region: should include switzerland (and analogously for European/EU → europe). Fix: decide which is right. If the body's footer is correct, soften the TL;DR phrasing (e.g. "EU/CH" → "EU"). If the TL;DR is correct, add switzerland to the body footer's Region: field (taxonomy already permits both regions on one item). Don't suppress — drift between the bullet and the body footer is reader-visible and reads as inconsistency.
  • name-collision WARNs (v2.52) — H3 items that share a proper noun with prior coverage and lack explicit disambiguation. Each WARN names the H3 heading prefix and the colliding candidate name (from work/<run-id>/prior_coverage.json's name_collision_candidates list). Fix: re-read the cited source. If today's item is genuinely a different entity sharing the same name (e.g. defender tool reusing an attacker codename, or vice versa — the canonical case is the Datadog/TeamPCP Shai-Hulud inversion), add an explicit disambiguation phrase to the body: "named for the attacker tooling", "no relation to the X campaign", "not to be confused with", "shares the name", "namesake". If today's item is the same entity referenced in prior coverage, the H3 should be an UPDATE: block linking back (the check auto-exempts UPDATEs). If neither applies, you have a name confusion that needs to be resolved before publish — re-read the source carefully; this WARN is the system telling you to verify whether you're describing the attacker or the defender. The 2026-05-15 cap-breach started here.
  • evidence-binding WARNs (v2.53) — quotes in an Evidence: field whose (Publisher) attribution does not match any of the item's Source: / Additional source: publisher labels. Fix: change the attribution to match the publisher label of the source the quote actually came from, or add the source to the footer if you forgot to cite it. The attribution is the only link between the quote and a fetched source the reader can verify — an unbound quote loses that link. WARN-level in v2.53 (the field is optional); v2.54+ may upgrade to FAIL once the noise floor is calibrated on historical briefs.

Operating principle

The script is read-only by design — drift is what you fix; the script just surfaces it. Maintaining tools/check_brief.py is part of the agent's self-evolution authority — when a new check would catch a class of drift slipping through, add it in the same run as the brief.

If tools/check_brief.py itself fails to start (SystemExit 2, import error, missing taxonomy), proceed to Phase 6 anyway and log the script-level error in § 8 — never let tooling block the brief. The CRITICAL anti-crash header at the top of the daily prompt always wins.