OPEN METHOD · DESIGNED BY HUANG SHENGZE

$tau2-result-audit-v1

A causal audit system for deciding why a τ²-Bench outcome happened—and whether that outcome actually represents Agent capability.

一套用于判断 τ²-Bench 结果为何发生、以及该结果能否代表 Agent 能力的因果审计系统。

01 · THE IDEA01 · 核心思路

Audit the measurement before blaming the model.

先审计测量,再判断模型。

The unit of analysis is the whole interactive system, not the final Agent action.

分析单位是完整交互系统,而不是最后一次 Agent 动作。

01

Reconstruct reality先还原事实

Turn hidden Scenario, visible Policy, effective Rubric, DB and dialogue into one consistent event timeline.

把隐藏 Scenario、可见 Policy、实际 Rubric、数据库和对话还原成同一条事件时间线。

02

Find causality再寻找因果

Find the earliest event after which Gold became unreachable, instead of blaming the last tool executor.

寻找最早让 Gold 不再可达的事件,而不是机械归责于最后的工具执行者。

03

Test the claim用反事实验证

Change only User, Agent, Environment or Rubric. The smallest repair that restores reachability identifies causal responsibility.

只修 User、Agent、Environment 或 Rubric 中的一方;能恢复可达性的最小修正用于识别责任。

02 · ARCHITECTURE02 · 审计架构

Evidence in. Causal verdict out.

输入证据,输出因果结论。

Four layers preserve the path from source fact to every Task-, model- and trial-level claim.

四层架构保留从源码事实到 Task、模型和逐轨迹结论的完整路径。

INPUT

Evidence plane证据层

  • Task + User Scenario
  • Policy + tools
  • reward_basis + Gold
  • Initial DB + Environment
  • All dialogues + tool traces
→
MODEL

Reconstruction plane还原层

  • Hidden event state machine隐藏事件状态机
  • Agent-visible information boundaryAgent 可见信息边界
  • Effective scoring function实际生效评分函数
  • Field-level DB transition字段级数据库变化
→
ENGINE

Causal plane因果层

  • Earliest decisive divergence最早决定性偏离
  • Failure clustering失败机制聚类
  • Single-variable counterfactual单变量反事实
  • Successful reachability control成功可达性对照
→
OUTPUT

Verdict plane结论层

  • User / Agent / Environment
  • Infrastructure / Mixed
  • Benchmark overlay
  • official_result + audit_result
  • Traceable evidence index可追溯证据索引
Direct executor直接执行者Who changed the DB?谁改变了数据库?
Behavioral cause行为责任者Who first made Gold unreachable?谁最先让 Gold 不再可达?
Benchmark validity基准有效性Does the score measure the real goal?分数是否测到了真实目标?

03 · DECISION LOGIC03 · 判断逻辑

One claim, one evidence chain.

一个结论,一条完整证据链。

  1. 01

    Establish the experiment matrix确认实验矩阵

    Domain, Task, Agent, User Simulator, Judge, trial, model parameters, missing or duplicated runs, and infrastructure failures.

    确认 domain、Task、Agent、User Simulator、Judge、trial、模型参数、缺失、重复和基础设施异常。

  2. 02

    Compile the Task into a state machine把 Task 编译成状态机

    Define initial intent, staged disclosure, observable triggers, valid next actions, equivalent Policy-legal paths and termination.

    定义初始意图、逐步披露、可观察触发器、合理下一步、Policy 等价合法路径和终止条件。

  3. 03

    Compute the effective Rubric计算实际 Rubric

    Only components in reward_basis determine Reward. Gold actions define a target state; displayed Judge text may be unscored.

    只有 reward_basis 中的分量决定 Reward。Gold actions 定义目标状态;报告里的 Judge 文字可能完全不计分。

  4. 04

    Diff the database and visible trace对齐数据库与可见轨迹

    Compare intended and actual writes field by field, while judging each participant only from information available at that moment.

    逐字段比较预期与实际写入,同时只按参与者当时真正可见的信息判断行为。

  5. 05

    Locate the earliest decisive divergence定位最早决定性偏离

    Cluster every failure by the first event that removed a passing path. The final wrong write is evidence, not automatically the cause.

    按最早移除通过路径的事件聚类全部失败。最终错误写入是证据,不自动等于根因。

  6. 06

    Run single-variable counterfactuals执行单变量反事实

    Ask whether correcting only User, Agent, Environment or Rubric restores Gold. If one repair is insufficient, classify independent causes as Mixed.

    判断只修 User、Agent、Environment 或 Rubric 是否恢复 Gold;只修一方仍不够时,才判为 Mixed。

  7. 07

    Audit success as a control把成功轨迹作为对照

    A reward-1 run must still satisfy User fidelity, Policy compliance, correct writes and real goal completion. Otherwise it is a false success.

    Reward 1 仍需满足 User 忠实、Policy 合规、写入正确和真实目标完成,否则是假成功。

  8. 08

    Publish two results保留两套结果

    official_result preserves leaderboard compatibility; audit_result records whether the trajectory is valid evidence of capability.

    official_result 保持排行榜兼容;audit_result 记录该轨迹是否真能作为能力证据。

04 · CLAIM THRESHOLD04 · 结论门槛

Case evidence is not Task evidence.

个案证据不等于 Task 级证据。

CASE LEVEL

One trajectory can prove单条轨迹可以证明

What happened in that run, or a direct deterministic contradiction between Task and Gold.

该次运行发生了什么,或 Task 与 Gold 之间确定性的直接矛盾。

TASK LEVEL

A complete audit requires完整审计必须覆盖

All trials, effective reward components, failure groups, a successful reachability control, independent responsibility checks and benchmark validity.

全部轨迹、实际计分分量、失败分组、成功可达性对照、独立责任检查和 Benchmark 有效性。

STRUCTURAL CLAIM

Stronger evidence requires结构性结论需要

Repeated divergence across models, a large shift after swapping one participant, or a contradiction intrinsic to Task, Gold or Rubric.

多个模型在同一节点偏离、只替换一个参与者后显著迁移,或 Task、Gold、Rubric 自身存在矛盾。

05 · PROOF BY USE05 · 在 τ²-Bench 中的实际作用

Four defects, four different causal shapes.

四个问题,四种不同因果结构。

06 · FULL PUBLIC SPECIFICATION06 · 完整公开原文

No private document required.

无需飞书权限即可阅读。

The complete original Chinese SKILL.md is published verbatim as a first-class site artifact.

完整中文 SKILL.md 已作为本站正式内容公开,不依赖飞书访问权限。

01Scope and Task-level evidence threshold审查范围与 Task 级结论门槛
02Task state-machine reconstructionTask 隐藏状态机还原
03Effective Rubric, Gold and DB audit实际 Rubric、Gold 与数据库审计
04All-trial coverage and success controls全部轨迹覆盖与成功对照
05Causal attribution and Agent responsibility因果归因与 Agent 独立责任审查
06Benchmark validity and controlled experimentsBenchmark 有效性与对照实验
07Traceable output schema and writing rules可追溯输出结构与写作要求