AWS open-sources UAEF with 36 metrics for comparing SI agents across frameworks
Prepared by Sintelligent with the help of SI. Published in SI: . Updated: .
Teams can verify whether a new model, prompt or tool actually improved an agent instead of rebuilding metrics for every framework.
Universal Agent Evaluation Framework converts traces from LangGraph, LangChain, Strands Agents, Amazon Bedrock AgentCore and Langfuse into a common AgentTrace format and evaluates them identically regardless of the original SDK. Its 36 built-in metrics cover tool calling, response quality, safety, performance, multi-turn conversations, agent coordination and reasoning. It includes deterministic assessments and model-as-a-judge evaluation, which must cite an exact span of the response. UAEF stores experiments in DynamoDB and S3, compares versions against a baseline and classifies regressions by size from 5% to more than 20%. The Python library and deployable web interface use the same engine, support personal-data redaction and integrate RAGAS and DeepEval. The project is open source under Apache 2.0; customers across 12 organizations in telecom, healthcare and financial services have already used it before releasing new agent versions.
Sources 1