evals repositories
Trending repositories tagged evals, ranked by trending score. Updated daily.
-
1
FailproofAI/failproofai
Observability and enforcement for AI agent harnesses. Capture every run and runtime reliability with policy enforcement.
GitHub · TypeScript · 9,877 stars · 580 forks · +218 stars in 24 hours · Trending score 4.22
-
2
harbor-framework/harbor
Framework for evaluating and improving agents
GitHub · Python · 7,264 stars · 2,250 forks · +134 stars in 24 hours · Trending score 4.01
-
3
mastra-ai/mastra
Mastra is the modern TypeScript framework for AI-powered applications and agents.
GitHub · TypeScript · 29,686 stars · 3,124 forks · +94 stars in 24 hours · Trending score 3.67
-
4
heymrun/heym
Build agentic systems. Run them with confidence. Orchestrate agents, automate business processes, inspect every execution, and keep humans in control. Deploy Heym on your own infrastructure.
GitHub · Python · 1,791 stars · 137 forks · +53 stars in 24 hours · Trending score 3.39
-
5
Arize-ai/phoenix
AI Observability & Evaluation
GitHub · Python · 12,363 stars · 1,319 forks · +26 stars in 24 hours · Trending score 3.00
-
6
ShenSeanChen/waku-agent
Waku Waku! Waku Agent is a local-first AI agent harness you actually own, including loop, memory, eval, all in code built to stay legible as it grows.
GitHub · Python · 2,205 stars · 437 forks · +25 stars in 24 hours · Trending score 2.65
-
7
pydantic/logfire
AI observability platform for production LLM and agent systems.
GitHub · Python · 4,561 stars · 376 forks · +6 stars in 24 hours · Trending score 1.90
-
8
rails/lemans
A Ruby harness for running agent benchmarks
GitHub · Ruby · 100 stars · 12 forks · +11 stars in 24 hours · Trending score 1.87
-
9
lmnr-ai/lmnr
Laminar - open-source observability platform purpose-built for AI agents. YC S24.
GitHub · TypeScript · 3,510 stars · 262 forks · +8 stars in 24 hours · Trending score 1.85
-
10
Valen-Team/Roller
Turn agent runs into training data. Roller rolls out Claude Code, Codex, OpenClaw and four other Harnesses in disposable Docker containers, grades benchmark tasks in place, and exports SFT-ready trajectories that keep reasoning, images and compacted-context segments.
GitHub · Python · 106 stars · 0 forks · +37 stars in 24 hours · Trending score 1.78
-
11
MCPJam/inspector
Testing and evaluation platform to chat, inspect, and debug MCP servers, MCP apps, and ChatGPT apps.
GitHub · TypeScript · 2,331 stars · 305 forks · +3 stars in 24 hours · Trending score 1.57
-
12
future-agi/future-agi
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.
GitHub · Python · 2,288 stars · 725 forks · +2 stars in 24 hours · Trending score 1.55
-
13
truera/trulens
Evaluation and Tracking for LLM Experiments and AI Agents
GitHub · Python · 3,659 stars · 397 forks · 0 stars in 24 hours · Trending score 1.28
-
14
overmind-core/overmind
The platform for continuously improving AI agents.
GitHub · Python · 766 stars · 76 forks · +4 stars in 24 hours · Trending score 1.26
-
15
jackulau/coding.sh
Open-source, local-first observability and evals for AI agents: see what Claude Code, Codex, Cursor and others did and what it cost.
GitHub · TypeScript · 117 stars · 4 forks · 0 stars in 24 hours · Trending score 1.25
-
16
Kiln-AI/Kiln
Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.
GitHub · Python · 5,286 stars · 406 forks · +1 stars in 24 hours · Trending score 1.23
-
17
LilMGenius/paperthin
The skillbook for recursive self-improvement (RSI). NVIDIA's latest proof, unpacked with timeless engineering. Distilled from the OpenAI harness engineering hackathon winner's zero-touch method.
GitHub · JavaScript · 1,448 stars · 123 forks · 0 stars in 24 hours · Trending score 1.23
-
18
awslabs/uaef
Universal Agent Evaluation Framework (UAEF) is a framework-agnostic evaluation system for AI agents. Invoke any agent and score it on multiple metrics spanning tool calling, response quality, safety, performance, and reasoning. Track experiments against baselines, catch regressions automatically, and get LLM-generated insights.
GitHub · Python · 93 stars · 8 forks · 0 stars in 24 hours · Trending score 1.22
-
19
github/evals
Explore, evaluate, and improve GitHub Copilot customizations with repeatable evals and saved results.
GitHub · 15 stars · 0 forks · +8 stars in 24 hours · Trending score 1.15
-
20
surus-lat/benchy
A benchmarking engine for evaluating AI systems on task-specific performance.
GitHub · Python · 12 stars · 0 forks · +3 stars in 24 hours · Trending score 1.02
-
21
ratel-ai/ratel
The self-improvement layer for AI agents. Ratel reads your agent's traces, finds what fails, recommends fixes and runs evals on real sessions before you ship. Self-hosted: one app, one Postgres. Includes the TypeScript and Python SDK
GitHub · TypeScript · 567 stars · 33 forks · +1 stars in 24 hours · Trending score 0.99
-
22
osolmaz/switch
A framework for building decision-model harnesses for any kind of task. Your program offers the model a short list of next steps, and the model picks one at a time until it can answer.
GitHub · TypeScript · 449 stars · 32 forks · +1 stars in 24 hours · Trending score 0.89
-
23
METR/hawk
Run Inspect AI evals in the cloud
GitHub · Python · 119 stars · 70 forks · 0 stars in 24 hours · Trending score 0.88
-
24
anhermon/awesome-agent-observability
Curated list of LLM and AI agent observability and evaluation tools: tracing, evals, LLM-as-judge, guardrails, OpenTelemetry GenAI, MCP.
GitHub · Python · 65 stars · 36 forks · +1 stars in 24 hours · Trending score 0.82