FIELD NOTES 001 · FOUR CASES · 48 TASK-LEVEL TRIALS 研究手记 001 · 四个案例 · 48 条任务级轨迹

When Agent Failures Are Benchmark Failures 当智能体失败其实是 基准测试失败

Built with Huang Shengze's $tau2-result-audit-v1 skill: reconstruct the hidden task, effective rubric, database transitions and full dialogue before assigning blame.

本文由黄胜泽设计的 $tau2-result-audit-v1 skill 驱动:先还原隐藏任务、实际评分规则、数据库变化与完整对话,再谈责任。

THE THESIS核心结论

Official reward is not causal truth.

官方奖励不是因果真相。

An interactive benchmark measures a coupled system. The user can drift, the task can contradict Gold, the environment can accept unrealistic writes, and the evaluator can ignore the behavior it claims to test.

交互式基准测试测量的是一个耦合系统:用户会漂移,任务会与 Gold 冲突,环境会接受不真实的写入,评分器也可能漏掉它声称要测的行为。

TASK×USER×AGENT×ENV×RUBRIC

The final reward multiplies only the components named in reward_basis. Everything else can be visible in a report and still have zero effect on score.

最终奖励只计算 reward_basis 指定的分量;其他检查即使出现在报告中,也可能完全不影响得分。

False positiveWrong behavior, reward 1行为错误,奖励为 1
False negativeValid path, reward 0路径合理,奖励为 0
Contaminated trialThe user changes the evaluated task用户模拟器改变了被评任务

$TAU2-RESULT-AUDIT-V1 · BY HUANG SHENGZE$TAU2-RESULT-AUDIT-V1 · 黄胜泽设计

From reward labels to causal evidence.

从奖励标签走向因果证据。

This is not a retrospective label: the skill generated the evidence chain used by every case below.

这不是事后贴标签:下方每个案例的证据链都由这套 skill 方法生成。

01

Reconstruct还原事实

Hidden Task state machine, Agent-visible Policy, effective Rubric, initial DB, Gold DB and full dialogue.

还原隐藏 Task 状态机、Agent 可见 Policy、实际 Rubric、初始与 Gold 数据库、完整对话。

02

Find divergence定位偏离

Locate the earliest event that made Gold unreachable—not the last tool call that exposed the damage.

寻找最早让 Gold 不再可达的事件,而不是只盯着最后暴露损害的工具调用。

03

Counterfactual反事实判断

Change only User, Agent, Environment or Rubric. Which single correction restores a passing path?

只修正 User、Agent、Environment 或 Rubric 中的一方,哪一项能恢复通过路径?

04

Audit success too成功也要审

A reward-1 trajectory is evidence only after user fidelity, policy compliance, writes and real task completion all hold.

只有用户忠实、政策合规、写操作正确且真实目标完成时,Reward 1 才能算成功证据。

V1 thresholdV1 结论门槛

A Task-level claim requires all trials, a successful-path control, effective reward components, failure clusters and independent responsibility checks. A single trajectory supports only a case-level claim unless it proves a direct Task–Gold contradiction.

Task 级结论必须覆盖全部轨迹、成功对照、实际计分分量、失败分组及独立责任检查。单条轨迹通常只能支持 case 级结论,除非它直接证明 Task 与 Gold 矛盾。

ORIGINAL AUDIT SKILL

$tau2-result-audit-v1

Designed by Huang Shengze to turn τ²-Bench run artifacts into causally defensible Task-, model- and trial-level conclusions.

由黄胜泽设计,用于把 τ²-Bench 运行产物转化为因果可辩护的 Task、模型与逐轨迹结论。

Public method, architecture and full specification →公开的方法、架构与完整原文 →

Its key contribution is to separate the direct tool executor, the first behavioral cause, and benchmark validity—then verify the conclusion with a one-variable counterfactual.

它的关键贡献是区分直接工具执行者、最早行为责任者与 Benchmark 有效性,再用单变量反事实验证结论。

WHY THIS SKILL FITS τ²-BENCH为什么这套 SKILL 特别适合 τ²-BENCH
τ²-Bench evidenceτ²-Bench 证据层 Skill operationSkill 处理方式 What it exposed here本次实际发现
Task + User Scenario Rebuild an event-driven hidden state machine重建事件驱动的隐藏状态机 Task 6 has no valid exit; Retail Task 5 uses an ambiguous confirmation triggerTask 6 没有合法退出;Retail Task 5 的确认触发器含糊
Policy + Tools Judge the Agent only from visible constraints只按 Agent 可见约束判断责任 Booking a duplicate was visible-policy legal even though hidden Gold forbade it重复订票符合可见 Policy,但隐藏 Gold 不允许
reward_basis + Gold Compute the rubric that actually affects reward计算真正影响 Reward 的 Rubric Task 6 excludes its policy assertion; Task 7 reduces meaning to a “1628” substringTask 6 排除了政策断言;Task 7 把语义退化成“1628”字符串
DB + Environment Compare actual writes with Gold field by field逐字段比较实际写入与 Gold Separated User-caused duplicate writes from correct environment execution区分 User 导致的重复写入与 Environment 的正确执行
Dialogue + Tool trace Locate the first decisive divergence定位最早决定性偏离 Responsibility moved from the final Agent write back to the earlier User goal change责任从最终 Agent 写操作回溯到更早的 User 目标漂移
All trials + Reward 1 Cluster every failure and audit successful controls聚类全部失败并审查成功对照 Proved Retail Task 5 is reachable, and Telecom is a true service success with an unscored defect证明 Retail Task 5 可达,也识别出 Telecom 是带未计分缺陷的服务真成功

FOUR CASE FILES四个深度案例

The actor is not always the cause.

直接执行者不一定是行为责任者。

01
TELECOMSOURCE VERIFIEDREWARD-1 AUDITED

Persona, database and simulated user disagree about one identity.

Persona、数据库与模拟用户给出了三套身份事实。

The Hard persona fixes age 64; the generated task binds it to John Smith, whose DOB is 1985-06-15. When asked for DOB, a Qwen user invented 1959-03-15. Yet the agent recovered through phone lookup, found an expired contract and correctly transferred the call.

Hard Persona 固定写着 64 岁;生成任务又把它绑定到出生于 1985-06-15 的 John Smith。Qwen 用户被问生日时编造了 1959-03-15,但 Agent 随后通过电话查询恢复,发现合同过期并正确转人工。

64-YEAR PERSONA→1985 DB→1959 INVENTED→PHONE RECOVERY→REWARD 1
Earliest divergence最早偏离Inconsistent benchmark facts precede dialogue对话前已有基准事实冲突
Counterfactual反事实Align persona age with account DOB让 Persona 年龄与账户生日一致
U · B overlay

True service success, plus a real unscored identity defect.

服务结果是真成功,同时存在未计分的身份缺陷。

Issue #492 ↗
02
AIRLINE · TASK 6SOURCE VERIFIED24 TRIALS AUDITED

An impossible goal and no exit condition push the user to invent a workaround.

不可完成的目标加上缺失的退出条件,诱导用户寻找绕行方案。

The user must persistently demand post-booking insurance and refuse transfer, while policy makes that goal impossible and the User Simulator must continue until completion. In 7/24 runs the user searched outside the original goal; one explicitly requested and twice confirmed a duplicate insured booking.

用户必须持续要求给已完成预订加保险并拒绝转人工,但 Policy 明确禁止事后加保险,全局 User Simulator 又要求继续到任务完成。24 条中有 7 条开始寻找原目标之外的绕行,1 条明确提出并两次确认创建重复保险订单。

7/24workaround drift出现绕行漂移
1/24duplicate DB write实际写入重复订单
0explicit exit rules明确退出规则
Earliest divergence最早偏离User changes goal from PEP4E0 to a new bookingUser 把目标从 PEP4E0 改成新建订单
Direct executor直接执行者Agent → book_reservation
Counterfactual反事实A faithful User alone restores Gold只让 User 忠实即可恢复 Gold
U primary

User is the behavioral cause; missing scope and exit rules are a Benchmark overlay.

User 是行为主因;缺少范围和退出规则构成 Benchmark 覆盖问题。

Issue #494 ↗
03
AIRLINE · TASK 7SOURCE VERIFIED12 TRIALS AUDITED

“Other upcoming flights” has two totals—but Gold permits only one.

“其他即将出发的航班”存在两个总价,但 Gold 只允许一个。

Gold requires $1,628, the pre-cancellation total of all four upcoming reservations. Natural language can exclude XEHM4B and 59XX6W—the two already under discussion—and produce $708 for the two “other” reservations. A substring evaluator searches every assistant message for 1628 without checking scope or timing.

Gold 要求回答取消前四张即将出发预订的总额 1,628 美元;但自然语义会排除正在讨论的 XEHM4B 与 59XX6W,得到两张“其他”预订合计 708 美元。评分器只在全部 Agent 文本中搜索 1628,不理解范围和时点。

ALL FOUR$1,628296 + 624 + 402 + 306
“OTHER” TWO$708402 + 306
9/12correct visible-message timing可见消息时点正确
8/12Benchmark semantic main causeBenchmark 语义主因
4/12Agent/User anomaly groupsAgent/User 异常分组
Earliest divergence最早偏离Task wording conflicts with Gold before any model acts任何模型行动前,Task 措辞已与 Gold 冲突
Counterfactual反事实Change only wording or target; valid trajectories pass只改措辞或目标值,合理轨迹即可通过
B primary

The dominant failure is measurement semantics, with smaller independent Agent and User groups.

主要失败来自测量语义,另有少量独立 Agent 与 User 异常。

Issue #495 ↗
04
RETAIL · TASK 5SOURCE VERIFIED12 TRIALS + CONTROL

A hidden two-step reversal is triggered by the ambiguous word “confirmation.”

隐藏的两阶段反悔,由含糊的“确认”一词触发。

Gold expects the user to change from two exchanges, to one exchange, then to no exchange and one return. But “if the agent asks for confirmation” could mean identity, order, variant, payment or the final write. In 11/12 runs the user skipped or delayed a required reversal; the agent then followed the user's visible “Yes.”

Gold 期待用户从交换两件改为一件,再改为不交换、只退水瓶;但“如果 Agent 要求确认”可能指身份、订单、Variant、支付或最终写操作。12 条中有 11 条 User 漏掉或延迟了必要反悔,Agent 随后只是执行了用户可见的“Yes”。

2 EXCHANGES1st write confirm1 EXCHANGE2nd write confirm1 RETURN
1/12true successful control完整成功对照
11/12User-primary failuresUser 主因失败
11 → 4failures after User swap只换 User 后失败数
Earliest divergence最早偏离User confirms a write instead of executing the hidden reversalUser 确认写操作,而未执行隐藏反悔
Agent responsibilityAgent 责任No clean Agent-primary failure in the 1111 条中没有干净的 Agent 主因
U primary

The task is reachable, but the score is dominated by User Simulator state-machine fidelity.

任务可达,但得分主要受 User Simulator 隐藏状态机忠实度支配。

Issue #496 ↗
OFFICIAL RESULT VS AUDIT RESULT官方结果与审计结果对照
Case Effective rubric实际 Rubric First decisive divergence最早决定性偏离 Direct executor直接执行者 Audit result审计结论
Telecom Reward 1 Persona–DB contradictionPersona–DB 冲突 User Sim True service success; unscored defect服务真成功;缺陷未计分
Airline 6 DB × empty COMMUNICATE User changes scopeUser 改写目标范围 Agent User primary; Benchmark overlayUser 主因;Benchmark 覆盖
Airline 7 DB × “1628” substring Task–Gold semantic conflictTask–Gold 语义冲突 — Benchmark primaryBenchmark 主因
Retail 5 DB × empty NL assertion User misses hidden reversalUser 漏掉隐藏反悔 Agent User primary; fidelity-sensitive TaskUser 主因;Task 高度依赖忠实度

CAUSAL ATTRIBUTION因果归因

Separate actor, cause and validity.

区分执行者、行为原因与测评有效性。

“Who called the write tool?” is different from “Who first made Gold unreachable?” Benchmark structure is an overlay: it can coexist with a User- or Agent-primary trajectory.

“谁调用了写工具”不同于“谁最先让 Gold 不可达”。Benchmark 是覆盖标签,可以与 User 或 Agent 主因同时存在。

Benchmark overlay Task, Gold, trigger wording, rubric or evaluator makes the result misleading. Task、Gold、触发措辞、Rubric 或评分器让结果失真。

REPAIR THE MEASUREMENT修复测量系统

Four changes make these tasks auditable.

四项修改,让任务真正可审计。

These are small data and evaluator changes, not a new platform.

这些是聚焦的数据与评分器修正,不需要另建平台。

01

Event triggers

Replace “first confirmation” and “third message” with observable events tied to user-visible replies and pending writes.

把“第一次确认”“第三条消息”改成与用户可见回复和待执行写操作绑定的可观察事件。

02

Exit conditions

Impossible goals need explicit refusal outcomes, scope limits and a valid STOP path.

不可完成目标必须写清拒绝后的结果、范围限制与合法 STOP 路径。

03

Scoped evaluators

Score the answer to a defined question at a defined time—not a number appearing anywhere in the transcript.

评分应绑定明确问题和明确时点,而不是在整段文本中搜索一个数字。

04

Split responsibility

Report Agent correctness, User fidelity and Benchmark validity separately before aggregating a leaderboard score.

先分别报告 Agent 正确性、User 忠实度和 Benchmark 有效性,再汇总排行榜分数。

REPORT MORE THAN ONE NUMBER不要只报告一个数字

A result needs its causal context.

每个结果都需要因果语境。

Sofficial

Leaderboard-compatible raw reward

兼容排行榜的官方原始奖励

Sagent

Only trials where User and Benchmark remain valid

仅统计 User 与 Benchmark 均有效的轨迹

Suser

User Simulator hidden-state fidelity

User Simulator 对隐藏状态机的忠实度

Sclean

Only tasks passing consistency checks and Gold replay

仅统计通过一致性检查与 Gold replay 的任务

EVIDENCE & LIMITS证据与限制

Source facts and run evidence are kept separate.

源码事实与实验观察分开陈述。

Verified on current main当前 main 已验证

All four Task definitions, policies, reward bases, database values and evaluator behavior.

四个 Task 的定义、Policy、reward basis、数据库值与评分器行为。

Observed in supplied audits用户实验中已观察

Task 6's 24-trial drift, Task 7's 12-trial causal groups, and Retail Task 5's 12-trial plus User-swap comparison.

Task 6 的 24 条漂移、Task 7 的 12 条因果分组,以及 Retail Task 5 的 12 条与换 User 对照。

Claim boundary结论边界

Trial counts describe the supplied runs. Deterministic Task–Gold contradictions support current-source claims; model-wide conclusions require larger matched experiments.

轨迹数量只描述现有实验;确定性的 Task–Gold 冲突支持源码级结论,模型级结论仍需更大规模匹配实验。