UiPath/coder_eval

README preview

# Coder Eval — evaluate & benchmark AI coding agents and Claude Code skills

[![PyPI](https://img.shields.io/pypi/v/coder-eval.svg)](https://pypi.org/project/coder-eval/)
[![GitHub Marketplace](https://img.shields.io/badge/marketplace-coder__eval-2ea44f.svg)](https://github.com/marketplace/actions/coder_eval)
[![Claude Code plugin](https://img.shields.io/badge/claude__code__plugin-coder--eval-d97757.svg)](docs/PLUGIN.md)
[![Website](https://img.shields.io/badge/website-coder--eval.com-1f6feb.svg)](https://coder-eval.com)
[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
[![Python 3.13+](https://img.shields.io/badge/python-3.13%2B-blue.svg)](https://www.python.org/downloads/)
[![CI](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml/badge.svg)](https://github.com/UiPath/coder_eval/actions/workflows/pr-checks.yml)

**Coder Eval** (`pip install coder-eval` / `uv tool install coder-eval`) is an open-source framework for
**evaluating and benchmarking AI coding agents and their skills** — built for CLI
and skill builders — with sandboxing, reproducibility, and data-driven analysis.
It runs a real agent (**Claude Code**, **Codex**, **Google Antigravity /
Gemini**, or **OpenCode**) in a sandbox against declarative YAML tasks, then scores the files and
commands it actually produced. Not an "agentic coding" benchmark: it measures how
effective your CLI and skills are when used by coding agents.

Reach for it when you want to **test whether a Claude Code skill triggers**,
**A/B-test Claude Code vs. Codex vs. Gemini vs. OpenCode** (or model vs. model,

View full repository on GitHub →