Skip to content

dsh-plugin-evaluation-standards

⭐ 1 · ✅ active · plugin
Type plugin Category Developer tools
Stars ⭐ 1 Status ✅ active
Author dsh-plugin-evaluation Updated 2026-08-20
Subcategory 🧪 Code, tests & review Capabilities coding

One-liner

Open evaluation datasets, test cases, and metrics for DSH plugins.

About

Each dataset is a profile (which metrics to use) and a cases file (test prompts and expected answers). Pick one that fits your plugin, run its cases, and use the results to understand how your plugin behaves.

✨ Key Features

  • Have a real scenario? Open an issue with how a user would ask, what t
  • Have a small set of cases? Submit a profile and cases following the contribution guide.
  • Maintain a dataset long term? Keep it in your own repository and add it to this catalog using the external dataset listing guide.

📦 Install

git clone --branch v1.1.0 --depth 1 \
  https://github.com/dsh-plugin-evaluation/dsh-plugin-evaluation-standards.git

🚀 Quick Start

profiles/<id>.json  Which metrics to use and where to find the cases
cases/<id>.json     Plugin types and test cases