Skip to content

Benchmark 批量评测 ​

概述 ​

benchmark_runner.py 提供了标准化的批量评测系统,支持:

  • 多轮次运行 (N episodes),每轮使用不同随机种子
  • 自动计算均值和标准差
  • 跨环境汇总统计
  • 结构化的评测报告输出

核心 API ​

run_benchmark() 函数 ​

python
from benchmark_runner import run_benchmark

result = run_benchmark(
    env_name="spread",           # 环境名 (9选1)
    provider="qwen",             # LLM 提供商
    episodes=5,                  # 运行轮数
    output_dir="results/benchmarks",  # 输出目录
    seed_start=1,                # 起始种子
    **game_kwargs                # 额外参数
)

参数说明:

参数类型默认值说明
env_namestr必填环境名称
providerstr必填LLM 提供商
episodesint3运行轮数
output_dirstr"results/benchmarks"输出目录
seed_startint1起始种子 (seed_start, seed_start+1, ...)

支持的环境名称:

python
GAME_RUNNERS = {
    "simple", "spread", "adversary", "push", "tag",
    "crypto", "reference", "speaker_listener", "world_comm"
}

run_single_episode() 函数 ​

运行单个 episode:

python
from benchmark_runner import run_single_episode
from pathlib import Path

stats = run_single_episode(
    env_name="spread",
    provider="qwen",
    episode_idx=1,
    output_dir=Path("results/benchmarks"),
    seed=42
)

输出结构 ​

文件输出 ​

results/benchmarks/
├── spread/
│   ├── spread_ep1.mp4          # Episode 1 游戏录像
│   ├── spread_ep1.json         # Episode 1 详细日志
│   ├── spread_ep2.mp4
│   ├── spread_ep2.json
│   └── ...
├── tag/
│   ├── tag_ep1.mp4
│   ├── tag_ep1.json
│   └── ...
└── ...

返回值结构 ​

python
result = {
    "env": "spread",
    "provider": "qwen",
    "episodes": 5,
    "mean_reward": -12.345,    # 跨 episode 平均奖励
    "std_reward": 2.456,       # 跨 episode 奖励标准差
    "episode_stats": [
        {
            "episode": 1,
            "env": "spread",
            "log": "results/benchmarks/spread/spread_ep1.json",
            "video": "results/benchmarks/spread/spread_ep1.mp4",
            "mean_reward": -11.2,
            "total_rewards": {
                "agent_0": -10.5,
                "agent_1": -12.1,
                "agent_2": -11.0
            },
            "steps": 25
        },
        // ...
    ]
}

使用示例 ​

单个环境快速测试 ​

python
result = run_benchmark(
    env_name="simple",
    provider="qwen",
    episodes=1
)
print(f"Mean Reward: {result['mean_reward']:.4f}")

全环境评测 ​

python
environments = [
    "simple", "spread", "adversary", "push", "tag",
    "crypto", "reference", "speaker_listener", "world_comm"
]

all_results = {}
for env_name in environments:
    result = run_benchmark(
        env_name=env_name,
        provider="qwen",
        episodes=5,
        output_dir="results/benchmarks",
    )
    all_results[env_name] = result
    print(f"{env_name}: {result['mean_reward']:.4f} ± {result['std_reward']:.4f}")

多模型对比 ​

python
providers = ["qwen", "deepseek", "openai"]

for provider in providers:
    result = run_benchmark(
        env_name="spread",
        provider=provider,
        episodes=10,
        output_dir=f"results/benchmarks/{provider}",
    )
    print(f"{provider}: {result['mean_reward']:.4f} ± {result['std_reward']:.4f}")

使用本地模型 ​

python
# Ollama
result = run_benchmark(
    env_name="tag",
    provider="ollama",
    episodes=5,
    model_name="qwen2.5:7b",
)

# Transformers
result = run_benchmark(
    env_name="spread",
    provider="transformers",
    episodes=3,
    model_path="Qwen/Qwen2.5-7B-Instruct",
    device="cuda",
)

日志解析 ​

JSON 日志格式 ​

每个 episode 的 JSON 日志包含:

json
[
  {
    "step": 0,
    "agent": "agent_0",
    "observation": { ... },
    "action": [0.0, 0.8, 0.0, 0.0, 0.5],
    "thought": "Target is left-up, applying thrust",
    "reward": -0.45
  },
  // ... 更多步骤
  {
    "final_summary": true,
    "total_rewards": { "agent_0": -12.3, "agent_1": -8.7 },
    "mean_reward": -10.5
  }
]

编程解析 ​

python
import json

with open("results/benchmarks/spread/spread_ep1.json") as f:
    data = json.load(f)

# 提取每步奖励
step_rewards = [entry["reward"] for entry in data if "step" in entry]

# 提取 LLM 思考过程
thoughts = [entry["thought"] for entry in data if "step" in entry]

# 提取最终统计
final = [entry for entry in data if entry.get("final_summary")]

统计分析 ​

run_benchmark() 自动计算两个核心统计指标:

指标公式含义
Mean RewardΣ(episode_mean) / N跨 episode 平均奖励
Std Reward√(Σ(x - mean)² / N)奖励的标准差

其中 episode_mean 是每个 episode 中所有智能体奖励的平均值。

Built with ❤️ for Multi-Agent AI Research