SkillEval

A Visual Workbench for AB Testing AI Skills

Description

How do you prove one AI prompt is better than another? Not vibes. Blind evaluation. SkillEval runs side-by-side comparisons of model outputs, scores them against automated criteria with the evaluator unable to see which version produced which result, and gives you a statistically grounded answer. Open source. I used it to rewrite Claude's frontend design skill and proved a 75% win rate over the original.

More context

SkillEval came out of a very practical annoyance: I had rewritten a Claude skill, but I needed a way to prove the new version was better than the old one. "Feels better" was not enough.

The workflow is deliberately direct. Upload two skills, define evaluation criteria, run them through the same prompts, and let an AI judge score the outputs without knowing which skill produced which answer. The result is not magic truth, but it is better than vibe-checking a prompt change in your head.

Later versions added support for zipped Agent Skill packages. That matters because real skills often have references, scripts, and assets, not a single Markdown file. SkillEval treats both paths as packages internally, so the test reflects how the skill actually runs.

Facts

Links

Media

Tags

JW
Justin Wetch