evaluation-framework

Test your prompts, agents, and RAGs. Red teaming/pentesting/vulnerability scanning for AI. Compare performance of GPT, Claude, Gemini, DeepSeek, and more. Simple declarative configs with command line and CI/CD integration. Used by OpenAI and Anthropic.

The LLM Evaluation Framework

A framework for few-shot evaluation of language models.

Build, Evaluate, and Optimize AI Systems. Includes evals, RAG, agents, fine-tuning, synthetic data generation, dataset management, MCP, and more.

Lighteval is your all-in-one toolkit for evaluating LLMs across multiple backends

☁️ 🚀 📊 📈 Evaluating state of the art in AI

Open-source, end-to-end platform for evaluating, observing, and improving LLM and AI agent applications. Tracing · Evals · Simulations · Datasets · Gateway · Guardrails. Self-hostable. Apache 2.0.

Repo for AI Agents The Definitive Guide

This is the repository of our article published in RecSys 2019 "Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches" and of several follow-up studies.

AgentLab: An open-source framework for developing, testing, and benchmarking web agents on diverse tasks, designed for scalability and reproducibility.

Data-Driven Evaluation for LLM-Powered Applications

PySODEvalToolkit: A Python-based Evaluation Toolbox for Salient Object Detection and Camouflaged Object Detection

Moonshot - A simple and modular tool to evaluate and red-team any LLM application.

Metrics to evaluate the quality of responses of your Retrieval Augmented Generation (RAG) applications.

Python SDK for running evaluations on LLM generated responses

MedEvalKit: A Unified Medical Evaluation Framework

The official evaluation suite and dynamic data release for MixEval.

build and benchmark deep research

A research library for automating experiments on Deep Graph Networks

AI Data Management & Evaluation Platform