llm-evaluation repositories

Trending repositories tagged llm-evaluation, ranked by trending score. Updated daily.

  1. 1
    jackson-cmd/risk-aware-trading-agents

    AI trading agents for US stocks with news backtesting that can't see the future: a public LLM trading agent leaderboard (rta-bench-1), frozen point-in-time 2020/2022/2026 markets (prices, VIX, news), a look-ahead check on every run, LLM agents over MCP, and paper/live trading on 10 brokers. 风险感知美股交易智能体 · 新闻回测 · LLM 榜单

    GitHub · Python · 355 stars · 80 forks · +101 stars in 24 hours · Trending score 3.90

  2. 2
    promptfoo/promptfoo

    Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

    GitHub · TypeScript · 27,512 stars · 2,808 forks · +64 stars in 24 hours · Trending score 3.79

  3. 3
    langfuse/langfuse

    🪢 Open source AI engineering platform: AI Gateway, agent observability, evals, and prompt/skill management, built on OpenTelemetry. One open platform for every AI call and every agent. Self-host or use Langfuse Cloud.

    GitHub · TypeScript · 37,691 stars · 4,285 forks · +76 stars in 24 hours · Trending score 3.79

  4. 4
    Continuum-AI-Corp/OrcaReplay

    OrcaReplay — Time travel for AI agents. Record, replay, fork, and debug any agent run with any model. Built by the OrcaRouter.ai team.

    GitHub · TypeScript · 1,074 stars · 172 forks · +49 stars in 24 hours · Trending score 3.74

  5. 5
    Tencent/AI-Infra-Guard

    A full-stack AI Red Teaming platform securing AI ecosystems via Agent Scan, Skills Scan, MCP scan, AI Infra scan and LLM jailbreak evaluation.

    GitHub · Python · 7,603 stars · 783 forks · +52 stars in 24 hours · Trending score 3.30

  6. 6
    mlflow/mlflow

    The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

    GitHub · Python · 28,920 stars · 6,649 forks · +41 stars in 24 hours · Trending score 3.26

  7. 7
    comet-ml/opik

    Debug, evaluate, and monitor your LLM applications, RAG systems, and agentic workflows with comprehensive tracing, automated evaluations, and production-ready dashboards.

    GitHub · Java · 23,478 stars · 1,967 forks · +24 stars in 24 hours · Trending score 2.91

  8. 8
    Arize-ai/phoenix

    AI Observability & Evaluation

    GitHub · Python · 12,367 stars · 1,319 forks · +19 stars in 24 hours · Trending score 2.85

  9. 9
    jackulau/coding.sh

    Open-source, local-first observability and evals for AI agents: see what Claude Code, Codex, Cursor and others did and what it cost.

    GitHub · TypeScript · 149 stars · 6 forks · +21 stars in 24 hours · Trending score 2.75

  10. 10
    wanghao-SunSky/cs329z-chinese

    斯坦福 Stanford CS329Z Engineering AI Agents 中文教程:面向 Python 开发者的 AI 智能体课程,19 讲与 4 项实践,涵盖 LLM、RAG、工具调用、记忆、多智能体、评测与安全。完整 Markdown、在线与离线 HTML。Chinese course notes.

    GitHub · HTML · 658 stars · 79 forks · +30 stars in 24 hours · Trending score 2.67

  11. 11
    AI45Lab/OpenART

    OpenART is an open-source framework designed to evaluate the safety and robustness of autonomous AI agents in dynamic, long-horizon, and stateful environments. It stress-tests agent runtimes against multi-step state poisoning, privilege escalation, and tool-use vulnerabilities across 10,000+ benchmark scenarios.

    GitHub · Python · 277 stars · 32 forks · +35 stars in 24 hours · Trending score 2.66

  12. 12
    ChenJiangxi/BaziQA

    555 real questions from 17 years (2010–2026) of the Global Fortune-Teller Contest, with BaZi & Zi Wei charts, scoring tools and human-champion scores. Paper: arXiv:2602.12889

    GitHub · Python · 82 stars · 19 forks · +26 stars in 24 hours · Trending score 2.47

  13. 13
    Future-Proof-DS/interactive-ai-ml-tools

    Free interactive tools for data scientists to learn MLOps, ML system design and agentic analytics: a production ML system map, ML project structure, data drift simulator, agentic analytics system map and AI evals.

    GitHub · 298 stars · 28 forks · +11 stars in 24 hours · Trending score 2.25

  14. 14
    Giskard-AI/giskard-oss

    🐢 Open-Source Evaluation & Testing library for LLM Agents

    GitHub · Python · 5,965 stars · 577 forks · +15 stars in 24 hours · Trending score 2.20

  15. 15
    ifixai-ai/iFixAi

    Independent Auditing of AI Agents. Run by human or the agent itself, to answer the most crucial question in the AI Agent Economy. Is the agent doing what is supposed to do? With iFixAi you can have this answer in less than 120 seconds.

    GitHub · Python · 46,553 stars · 1,828 forks · −189 stars in 24 hours · Trending score 2.05

  16. 16
    bottomless/agent-duel

    A desktop app for head-to-head coding agent battles, with single-agent mode and free inference. Pick the winning agent and continue from its work.

    GitHub · TypeScript · 70 stars · 22 forks · +5 stars in 24 hours · Trending score 1.96

  17. 17
    GenAI-Security-Project/crosswalk

    GitHub · JavaScript · 38 stars · 17 forks · +12 stars in 24 hours · Trending score 1.95

  18. 18
    coryparrry/Intents

    Intents is a native macOS workbench for evaluating Apple's on-device Foundation Models.

    GitHub · Swift · 19 stars · 0 forks · +13 stars in 24 hours · Trending score 1.92

  19. 19
    future-agi/future-agi

    Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.

    GitHub · Python · 2,285 stars · 725 forks · +4 stars in 24 hours · Trending score 1.78

  20. 20
    NVIDIA/garak

    the LLM vulnerability scanner

    GitHub · Python · 10,091 stars · 1,491 forks · 0 stars in 24 hours · Trending score 1.67

  21. 21
    overmind-core/overmind

    The platform for continuously improving AI agents.

    GitHub · Python · 772 stars · 76 forks · +9 stars in 24 hours · Trending score 1.66

  22. 22
    Q00/ouroboros

    Agent OS: the agent gets smarter on its own. We just hold the line: Interview-gated, staged evaluation, budgeted evolution loop. MCP server, 14 runtimes: Claude Code, Codex CLI, Gemini CLI, OpenCode, Copilot, Kiro and more.

    GitHub · Python · 6,386 stars · 635 forks · +5 stars in 24 hours · Trending score 1.60

  23. 23
    lmnr-ai/lmnr

    Laminar - open-source observability platform purpose-built for AI agents. YC S24.

    GitHub · TypeScript · 3,515 stars · 262 forks · +5 stars in 24 hours · Trending score 1.58

  24. 24
    Goldziher/ai-rulez

    Author, generate, validate and govern AI agent rules, skills, plugins and MCP configuration across 52 coding assistants. Review with verified model evidence, evaluate skills, lock and sign dependencies, and publish bundles. CLI, MCP server and Go API.

    GitHub · Go · 180 stars · 19 forks · +4 stars in 24 hours · Trending score 1.43