跳转至

dsh-eval
类型 工作流 分类 研究
星数 – 状态 ✅ 活跃
能力 research, workflow, observability 更新时间 2026-08-14

📖 项目简介

Agent Evaluation Platform for deepseek-harness. Run benchmarks against headless dsh profiles, harvest persisted session logs as traces, fold automatic metrics, grade task success and tool selection, and report or compare runs — one benchmark.yaml in, one JSON run + Markdown report out.

✨ 核心特性

  • dsh eval run benchmark.yaml — orchestrate one headless dsh subprocess per case × trial
  • Trace harvesting from persisted session logs (everything a model sees is reconstructable from the log)
  • Automatic metrics: task success, tool success, tool-selection accuracy, steps, tokens, latency, cost, retry, invalid tool calls, context usa
  • Scripted grading: expected.tool (tool-selection accuracy) and expected.check (task success)

📦 安装

pnpm add dsh-eval
dsh plugin --profile eval add dsh-eval

🚀 快速开始

git clone https://github.com/hccccc01333/dsh-eval.git
cd dsh-eval

🔗 链接