LLM Evals & Testing
Evaluation frameworks and benchmarks for measuring prompt, RAG and agent quality with repeatable tests, so regressions show up in CI instead of production.
-
BraintrustEvaluation-first platform for LLM applications, treating prompt and model changes like code changes that need regression tests before shipping.
-
Arize Phoenix
Open-source LLM observability and evaluation platform built on OpenTelemetry, with tracing, datasets, experiments and prompt tooling.
-
Opik
Comet's open-source platform for logging traces, running evaluations and monitoring LLM apps and agents in development and production.
-
Promptfoo
CLI and library for testing prompts and models, red-teaming and evaluating LLM apps.
-
DeepEval
Open-source LLM evaluation framework with unit-test style metrics for RAG and agents.
-
Ragas
Evaluation toolkit for RAG pipelines and LLM applications with automated quality metrics.
-
garak
NVIDIA's LLM vulnerability scanner that probes models for jailbreaks, leaks and other failures.
-
Inspect
Open-source framework from the UK AI Security Institute for running LLM and agent evaluations.
-
LM Evaluation Harness
EleutherAI's standard framework for benchmarking language models across hundreds of tasks.
-
SWE-bench
Benchmark that tests whether AI agents can resolve real GitHub issues in open-source repositories.
-
Galileo
Evaluation and observability platform for LLM apps and agents, with hallucination detection and guardrails.