llm-evaluation repositories
Trending repositories tagged llm-evaluation, ranked by trending score. Updated daily.
-
1
jackson-cmd/risk-aware-trading-agents
AI trading agents for US stocks with news backtesting that can't see the future: a public LLM trading agent leaderboard (rta-bench-1), frozen point-in-time 2020/2022/2026 markets (prices, VIX, news), a look-ahead check on every run, LLM agents over MCP, and paper/live trading on 10 brokers. 风险感知美股交易智能体 · 新闻回测 · LLM 榜单
GitHub · Python · 355 stars · 80 forks · +101 stars in 24 hours · Trending score 3.90
-
2
promptfoo/promptfoo
Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.
GitHub · TypeScript · 27,512 stars · 2,808 forks · +64 stars in 24 hours · Trending score 3.79
-
3
langfuse/langfuse
🪢 Open source AI engineering platform: AI Gateway, agent observability, evals, and prompt/skill management, built on OpenTelemetry. One open platform for every AI call and every agent. Self-host or use Langfuse Cloud.
GitHub · TypeScript · 37,691 stars · 4,285 forks · +76 stars in 24 hours · Trending score 3.79
-
4
Continuum-AI-Corp/OrcaReplay
OrcaReplay — Time travel for AI agents. Record, replay, fork, and debug any agent run with any model. Built by the OrcaRouter.ai team.
GitHub · TypeScript · 1,074 stars · 172 forks · +49 stars in 24 hours · Trending score 3.74
-
5
Tencent/AI-Infra-Guard
A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation.
GitHub · Python · 7,603 stars · 783 forks · +52 stars in 24 hours · Trending score 3.30
-
6
mlflow/mlflow
The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.
GitHub · Python · 28,920 stars · 6,649 forks · +41 stars in 24 hours · Trending score 3.26
-
7
comet-ml/opik
Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.
GitHub · Java · 23,478 stars · 1,967 forks · +24 stars in 24 hours · Trending score 2.91
-
8
Arize-ai/phoenix
AI Observability & Evaluation
GitHub · Python · 12,367 stars · 1,319 forks · +19 stars in 24 hours · Trending score 2.85
-
9
jackulau/coding.sh
Open-source, local-first observability and evals for AI agents: see what Claude Code, Codex, Cursor and others did and what it cost.
GitHub · TypeScript · 149 stars · 6 forks · +21 stars in 24 hours · Trending score 2.75
-
10
wanghao-SunSky/cs329z-chinese
斯坦福 Stanford CS329Z Engineering AI Agents 中文教程:面向 Python 开发者的 AI 智能体课程,19 讲与 4 项实践,涵盖 LLM、RAG、工具调用、记忆、多智能体、评测与安全。完整 Markdown、在线与离线 HTML。Chinese course notes.
GitHub · HTML · 658 stars · 79 forks · +30 stars in 24 hours · Trending score 2.67
-
11
AI45Lab/OpenART
OpenART is an open-source framework designed to evaluate the safety and robustness of autonomous AI agents in dynamic, long-horizon, and stateful environments. It stress-tests agent runtimes against multi-step state poisoning, privilege escalation, and tool-use vulnerabilities across 10,000+ benchmark scenarios.
GitHub · Python · 277 stars · 32 forks · +35 stars in 24 hours · Trending score 2.66
-
12
ChenJiangxi/BaziQA
555 real questions from 17 years (2010–2026) of the Global Fortune-Teller Contest, with BaZi & Zi Wei charts, scoring tools and human-champion scores. Paper: arXiv:2602.12889
GitHub · Python · 82 stars · 19 forks · +26 stars in 24 hours · Trending score 2.47
-
13
Future-Proof-DS/interactive-ai-ml-tools
Free interactive tools for data scientists to learn MLOps, ML system design and agentic analytics: a production ML system map, ML project structure, data drift simulator, agentic analytics system map and AI evals.
GitHub · 298 stars · 28 forks · +11 stars in 24 hours · Trending score 2.25
-
14
Giskard-AI/giskard-oss
🐢 Open-Source Evaluation & Testing library for LLM Agents
GitHub · Python · 5,965 stars · 577 forks · +15 stars in 24 hours · Trending score 2.20
-
15
ifixai-ai/iFixAi
Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.
GitHub · Python · 46,553 stars · 1,828 forks · −189 stars in 24 hours · Trending score 2.05
-
16
bottomless/agent-duel
A desktop app for head-to-head coding agent battles, with single-agent mode and free inference. Pick the winning agent and continue from its work.
GitHub · TypeScript · 70 stars · 22 forks · +5 stars in 24 hours · Trending score 1.96
-
17
GenAI-Security-Project/crosswalk
GitHub · JavaScript · 38 stars · 17 forks · +12 stars in 24 hours · Trending score 1.95
-
18
coryparrry/Intents
Intents is a native macOS workbench for evaluating Apple's on-device Foundation Models.
GitHub · Swift · 19 stars · 0 forks · +13 stars in 24 hours · Trending score 1.92
-
19
future-agi/future-agi
Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.
GitHub · Python · 2,285 stars · 725 forks · +4 stars in 24 hours · Trending score 1.78
-
20
NVIDIA/garak
the LLM vulnerability scanner
GitHub · Python · 10,091 stars · 1,491 forks · 0 stars in 24 hours · Trending score 1.67
-
21
overmind-core/overmind
The platform for continuously improving AI agents.
GitHub · Python · 772 stars · 76 forks · +9 stars in 24 hours · Trending score 1.66
-
22
Q00/ouroboros
Agent OS: the agent gets smarter on its own. We just hold the line: Interview-gated, staged evaluation, budgeted evolution loop. MCP server, 14 runtimes: Claude Code, Codex CLI, Gemini CLI, OpenCode, Copilot, Kiro and more.
GitHub · Python · 6,386 stars · 635 forks · +5 stars in 24 hours · Trending score 1.60
-
23
lmnr-ai/lmnr
Laminar - open-source observability platform purpose-built for AI agents. YC S24.
GitHub · TypeScript · 3,515 stars · 262 forks · +5 stars in 24 hours · Trending score 1.58
-
24
Goldziher/ai-rulez
Author, generate, validate and govern AI agent rules, skills, plugins and MCP configuration across 52 coding assistants. Review with verified model evidence, evaluate skills, lock and sign dependencies, and publish bundles. CLI, MCP server and Go API.
GitHub · Go · 180 stars · 19 forks · +4 stars in 24 hours · Trending score 1.43