Reconstruct还原事实
Hidden Task state machine, Agent-visible Policy, effective Rubric, initial DB, Gold DB and full dialogue.
还原隐藏 Task 状态机、Agent 可见 Policy、实际 Rubric、初始与 Gold 数据库、完整对话。
FIELD NOTES 001 · FOUR CASES · 48 TASK-LEVEL TRIALS 研究手记 001 · 四个案例 · 48 条任务级轨迹
Built with Huang Shengze's $tau2-result-audit-v1 skill: reconstruct the hidden task, effective rubric, database transitions and full dialogue before assigning blame.
本文由黄胜泽设计的 $tau2-result-audit-v1 skill 驱动:先还原隐藏任务、实际评分规则、数据库变化与完整对话,再谈责任。
THE THESIS核心结论
An interactive benchmark measures a coupled system. The user can drift, the task can contradict Gold, the environment can accept unrealistic writes, and the evaluator can ignore the behavior it claims to test.
交互式基准测试测量的是一个耦合系统:用户会漂移,任务会与 Gold 冲突,环境会接受不真实的写入,评分器也可能漏掉它声称要测的行为。
The final reward multiplies only the components named in reward_basis. Everything else can be visible in a report and still have zero effect on score.
最终奖励只计算 reward_basis 指定的分量;其他检查即使出现在报告中,也可能完全不影响得分。
$TAU2-RESULT-AUDIT-V1 · BY HUANG SHENGZE$TAU2-RESULT-AUDIT-V1 · 黄胜泽设计
This is not a retrospective label: the skill generated the evidence chain used by every case below.
这不是事后贴标签:下方每个案例的证据链都由这套 skill 方法生成。
Hidden Task state machine, Agent-visible Policy, effective Rubric, initial DB, Gold DB and full dialogue.
还原隐藏 Task 状态机、Agent 可见 Policy、实际 Rubric、初始与 Gold 数据库、完整对话。
Locate the earliest event that made Gold unreachable—not the last tool call that exposed the damage.
寻找最早让 Gold 不再可达的事件,而不是只盯着最后暴露损害的工具调用。
Change only User, Agent, Environment or Rubric. Which single correction restores a passing path?
只修正 User、Agent、Environment 或 Rubric 中的一方,哪一项能恢复通过路径?
A reward-1 trajectory is evidence only after user fidelity, policy compliance, writes and real task completion all hold.
只有用户忠实、政策合规、写操作正确且真实目标完成时,Reward 1 才能算成功证据。
A Task-level claim requires all trials, a successful-path control, effective reward components, failure clusters and independent responsibility checks. A single trajectory supports only a case-level claim unless it proves a direct Task–Gold contradiction.
Task 级结论必须覆盖全部轨迹、成功对照、实际计分分量、失败分组及独立责任检查。单条轨迹通常只能支持 case 级结论,除非它直接证明 Task 与 Gold 矛盾。
$tau2-result-audit-v1Designed by Huang Shengze to turn τ²-Bench run artifacts into causally defensible Task-, model- and trial-level conclusions.
由黄胜泽设计,用于把 τ²-Bench 运行产物转化为因果可辩护的 Task、模型与逐轨迹结论。
Public method, architecture and full specification →公开的方法、架构与完整原文 →Its key contribution is to separate the direct tool executor, the first behavioral cause, and benchmark validity—then verify the conclusion with a one-variable counterfactual.
它的关键贡献是区分直接工具执行者、最早行为责任者与 Benchmark 有效性,再用单变量反事实验证结论。
| τ²-Bench evidenceτ²-Bench 证据层 | Skill operationSkill 处理方式 | What it exposed here本次实际发现 |
|---|---|---|
| Task + User Scenario | Rebuild an event-driven hidden state machine重建事件驱动的隐藏状态机 | Task 6 has no valid exit; Retail Task 5 uses an ambiguous confirmation triggerTask 6 没有合法退出;Retail Task 5 的确认触发器含糊 |
| Policy + Tools | Judge the Agent only from visible constraints只按 Agent 可见约束判断责任 | Booking a duplicate was visible-policy legal even though hidden Gold forbade it重复订票符合可见 Policy,但隐藏 Gold 不允许 |
| reward_basis + Gold | Compute the rubric that actually affects reward计算真正影响 Reward 的 Rubric | Task 6 excludes its policy assertion; Task 7 reduces meaning to a “1628” substringTask 6 排除了政策断言;Task 7 把语义退化成“1628”字符串 |
| DB + Environment | Compare actual writes with Gold field by field逐字段比较实际写入与 Gold | Separated User-caused duplicate writes from correct environment execution区分 User 导致的重复写入与 Environment 的正确执行 |
| Dialogue + Tool trace | Locate the first decisive divergence定位最早决定性偏离 | Responsibility moved from the final Agent write back to the earlier User goal change责任从最终 Agent 写操作回溯到更早的 User 目标漂移 |
| All trials + Reward 1 | Cluster every failure and audit successful controls聚类全部失败并审查成功对照 | Proved Retail Task 5 is reachable, and Telecom is a true service success with an unscored defect证明 Retail Task 5 可达,也识别出 Telecom 是带未计分缺陷的服务真成功 |
FOUR CASE FILES四个深度案例
The Hard persona fixes age 64; the generated task binds it to John Smith, whose DOB is 1985-06-15. When asked for DOB, a Qwen user invented 1959-03-15. Yet the agent recovered through phone lookup, found an expired contract and correctly transferred the call.
Hard Persona 固定写着 64 岁;生成任务又把它绑定到出生于 1985-06-15 的 John Smith。Qwen 用户被问生日时编造了 1959-03-15,但 Agent 随后通过电话查询恢复,发现合同过期并正确转人工。
True service success, plus a real unscored identity defect.
服务结果是真成功,同时存在未计分的身份缺陷。
Issue #492 ↗The user must persistently demand post-booking insurance and refuse transfer, while policy makes that goal impossible and the User Simulator must continue until completion. In 7/24 runs the user searched outside the original goal; one explicitly requested and twice confirmed a duplicate insured booking.
用户必须持续要求给已完成预订加保险并拒绝转人工,但 Policy 明确禁止事后加保险,全局 User Simulator 又要求继续到任务完成。24 条中有 7 条开始寻找原目标之外的绕行,1 条明确提出并两次确认创建重复保险订单。
User is the behavioral cause; missing scope and exit rules are a Benchmark overlay.
User 是行为主因;缺少范围和退出规则构成 Benchmark 覆盖问题。
Issue #494 ↗
Gold requires $1,628, the pre-cancellation total of all four upcoming reservations. Natural language can exclude XEHM4B and 59XX6W—the two already under discussion—and produce $708 for the two “other” reservations. A substring evaluator searches every assistant message for 1628 without checking scope or timing.
Gold 要求回答取消前四张即将出发预订的总额 1,628 美元;但自然语义会排除正在讨论的 XEHM4B 与 59XX6W,得到两张“其他”预订合计 708 美元。评分器只在全部 Agent 文本中搜索 1628,不理解范围和时点。
The dominant failure is measurement semantics, with smaller independent Agent and User groups.
主要失败来自测量语义,另有少量独立 Agent 与 User 异常。
Issue #495 ↗Gold expects the user to change from two exchanges, to one exchange, then to no exchange and one return. But “if the agent asks for confirmation” could mean identity, order, variant, payment or the final write. In 11/12 runs the user skipped or delayed a required reversal; the agent then followed the user's visible “Yes.”
Gold 期待用户从交换两件改为一件,再改为不交换、只退水瓶;但“如果 Agent 要求确认”可能指身份、订单、Variant、支付或最终写操作。12 条中有 11 条 User 漏掉或延迟了必要反悔,Agent 随后只是执行了用户可见的“Yes”。
The task is reachable, but the score is dominated by User Simulator state-machine fidelity.
任务可达,但得分主要受 User Simulator 隐藏状态机忠实度支配。
Issue #496 ↗| Case | Effective rubric实际 Rubric | First decisive divergence最早决定性偏离 | Direct executor直接执行者 | Audit result审计结论 |
|---|---|---|---|---|
| Telecom | Reward 1 | Persona–DB contradictionPersona–DB 冲突 | User Sim | True service success; unscored defect服务真成功;缺陷未计分 |
| Airline 6 | DB × empty COMMUNICATE | User changes scopeUser 改写目标范围 | Agent | User primary; Benchmark overlayUser 主因;Benchmark 覆盖 |
| Airline 7 | DB × “1628” substring | Task–Gold semantic conflictTask–Gold 语义冲突 | — | Benchmark primaryBenchmark 主因 |
| Retail 5 | DB × empty NL assertion | User misses hidden reversalUser 漏掉隐藏反悔 | Agent | User primary; fidelity-sensitive TaskUser 主因;Task 高度依赖忠实度 |
CAUSAL ATTRIBUTION因果归因
“Who called the write tool?” is different from “Who first made Gold unreachable?” Benchmark structure is an overlay: it can coexist with a User- or Agent-primary trajectory.
“谁调用了写工具”不同于“谁最先让 Gold 不可达”。Benchmark 是覆盖标签,可以与 User 或 Agent 主因同时存在。
REPAIR THE MEASUREMENT修复测量系统
These are small data and evaluator changes, not a new platform.
这些是聚焦的数据与评分器修正,不需要另建平台。
Replace “first confirmation” and “third message” with observable events tied to user-visible replies and pending writes.
把“第一次确认”“第三条消息”改成与用户可见回复和待执行写操作绑定的可观察事件。
Impossible goals need explicit refusal outcomes, scope limits and a valid STOP path.
不可完成目标必须写清拒绝后的结果、范围限制与合法 STOP 路径。
Score the answer to a defined question at a defined time—not a number appearing anywhere in the transcript.
评分应绑定明确问题和明确时点,而不是在整段文本中搜索一个数字。
Report Agent correctness, User fidelity and Benchmark validity separately before aggregating a leaderboard score.
先分别报告 Agent 正确性、User 忠实度和 Benchmark 有效性,再汇总排行榜分数。
REPORT MORE THAN ONE NUMBER不要只报告一个数字
Leaderboard-compatible raw reward
兼容排行榜的官方原始奖励
Only trials where User and Benchmark remain valid
仅统计 User 与 Benchmark 均有效的轨迹
User Simulator hidden-state fidelity
User Simulator 对隐藏状态机的忠实度
Only tasks passing consistency checks and Gold replay
仅统计通过一致性检查与 Gold replay 的任务
EVIDENCE & LIMITS证据与限制
All four Task definitions, policies, reward bases, database values and evaluator behavior.
四个 Task 的定义、Policy、reward basis、数据库值与评分器行为。
Task 6's 24-trial drift, Task 7's 12-trial causal groups, and Retail Task 5's 12-trial plus User-swap comparison.
Task 6 的 24 条漂移、Task 7 的 12 条因果分组,以及 Retail Task 5 的 12 条与换 User 对照。
Trial counts describe the supplied runs. Deterministic Task–Gold contradictions support current-source claims; model-wide conclusions require larger matched experiments.
轨迹数量只描述现有实验;确定性的 Task–Gold 冲突支持源码级结论,模型级结论仍需更大规模匹配实验。