llm-evaluation-toolkit

Deliver safe & effective language models

Python SDK for running evaluations on LLM generated responses

A simple GPT-based evaluation tool for multi-aspect, interpretable assessment of LLMs.

Python SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

An easy python package to run quick basic QA evaluations. This package includes standardized QA evaluation metrics and semantic evaluation metrics: Black-box and Open-Source large language model prompting and evaluation, exact match, F1 Score, PEDANT semantic match, transformer match. Our package also supports prompting OPENAI and Anthropic API.

[ACL'25 Findings] Official repo for "HumanEval Pro and MBPP Pro: Evaluating Large Language Models on Self-invoking Code Generation Task"

[ACL'24] A Knowledge-grounded Interactive Evaluation Framework for Large Language Models

Core engine behind Calibrate, a framework for evaluating AI agents: speech-to-text, text-to-speech, LLM evaluation, end-to-end simulations

The E2E AI testing tool | No ML Overhead

scalexi is a versatile open-source Python library, optimized for Python 3.11+, focuses on facilitating low-code development and fine-tuning of diverse Large Language Models (LLMs).

Template for an AI application that extracts the job information from a job description using openAI functions and langchain

Measure of estimated confidence for non-hallucinative nature of outputs generated by Transformer-based Language Models.

Tools for systematic large language model evaluations

[EMNLP 2025 Findgings] When Punctuation Matters: A Large-Scale Comparison of Prompt Robustness Methods for LLMs.

TypeScript SDK for experimenting, testing, evaluating & monitoring LLM-powered applications - Parea AI (YC S23)

Frontend for Calibrate, a framework for evaluating AI agents: speech-to-text, text-to-speech, LLM evaluation, end-to-end simulations

Objectively test Claude SKILL.md files with blind A/B testing and a randomized judge. Catch regressions before they ship. CI-ready.

An epistemic AI lab for scientifically evaluating how language models track belief, truth, knowledge, inference, defeaters, and luck.

LLM-based chatbot using RAG to guide people around the KIT campus using natural language

Live evaluation harness for AI agents. Measure success rate and consistency. Lightweight local-first Python tool for testing Claude, GPT, Grok and custom agents.