bigcodebench

Official implementation of Learning What to Remember: Resource-Efficient Self-Improvement for Lifelong Agents. TokenWorth learns memory worth from execution feedback.

Benchmark suite for evaluating LLMs and SLMs on coding and SE tasks. Features HumanEval, MBPP, SWE-bench, and BigCodeBench with an interactive Streamlit UI. Supports cloud APIs (OpenAI, Anthropic, Google) and local models via Ollama. Tracks pass rates, latency, token usage, and costs.

Factor(UT): Controlling Untrusted AI Decomposers — AAAI 2026 workshop paper on monitoring untrusted decomposition in code generation workflows.

Resample or reroute after a weak-verifier stop? Pre-registered measurements of recoverable stopping debt on MBPP+, a two-sided action-support gate on BigCodeBench that fails closed, a LiveCodeBench observability ladder, and the exchangeable-actions reference showing realized-maximum gaps carry no selector signal. Artifacts for arXiv:2607.08665v3.

Research code and artifacts for How LLMs Fail in Code Generation: a cross-benchmark, fine-grained error taxonomy (SDDT 2026).

Federation-MOA: Lightweight Multi-Agent Sovereign Operating System & 3D World Tree Topology — lightweight 3-model workflow beats single frontier model (57.5% vs 45% on BigCodeBench-Hard). Pure stdlib, zero-dependency core.

Semantic deep-equality for BigCodeBench floating-point outputs. Published on PyPI.