Inicio / Skills / agent-evals-and-observability

agent-evals-and-observability

Provides framework- and vendor-neutral methods for evaluating AI agents, reviewing trajectories, analyzing regressions, setting release gates, and designing privacy-aware production observability.

★ 88 GitHub Stars

96 Puntuación de calidad

24 Archivos

Sobre esta Skill

Design, run, review, or release framework- and vendor-neutral evaluations and observability for AI agents. Use when defining agent evals, datasets, graders, trajectory review, regression analysis, release gates, production traces, or privacy-aware telemetry. Covers task and trajectory contracts, statistical comparisons, and incident-to-case learning; route framework implementation to pydanticai or langgraph when needed. Do not use this skill for unrelated requests; route to the nearest named specialist.

Casos de uso

Define task and trajectory contracts
Build evaluation datasets and graders
Compare agent versions statistically
Review traces, tools, and side effects
Design production telemetry and release gates

Funciones

Dataset manifests and evaluation plans
Deterministic, human, model, and domain grading guidance
Trajectory and trace review methods
Statistical comparison and regression analysis guidance
Privacy-aware observability recommendations

Agents compatibles

No especificado

Archivos de Skill

Other


                                evals/evals.json

JSON Other


                                README.md

Markdown Other


                                templates/dataset-manifest.md

Markdown Other


                                templates/eval-plan.md

Markdown Other


                                templates/grader-specification.md

Markdown Other


                                templates/metric-mirror.md

Markdown Other


                                templates/release-gate.md

Markdown Other


                                templates/run-report.md

Markdown Other


                                templates/trace-review.md

Markdown Reference


                                references/datasets.md

Markdown Reference


                                references/evaluation-design.md

Markdown Reference


                                references/evidence-modes-and-metric-mirrors.md

Markdown Reference


                                references/framework-routing.md

Markdown Reference


                                references/graders.md

Markdown Reference


                                references/grounding-attribution.md

Markdown Reference


                                references/judge-stability.md

Markdown Reference


                                references/metrics-and-statistics.md

Markdown Reference


                                references/opentelemetry-genai.md

Markdown Reference


                                references/production-observability.md

Markdown Reference


                                references/regression-and-release.md

Markdown Reference


                                references/source-index.md

Markdown Reference


                                references/synthetic-scenarios.md

Markdown Reference


                                references/trajectory-review.md

Markdown SKILL.md


                                SKILL.md

Markdown

Skills relacionadas

Pruebas 96

qa-methodology

Provides software teams with guidance, templates, references, and Python scripts for test strategy, regression, risk-based testing, CI triage, quality gates, exploratory testing, …

magnus919/agent-skills ★ 88

quality assurance test strategy test automation regression testing

Pruebas 96

playwright

Enables agents to author, run, debug, and triage Playwright end-to-end tests, integrate suites with CI, perform headless browsing and scraping, and check accessibility snapshots.

magnus919/agent-skills ★ 88

playwright e2e testing browser automation ci

Pruebas 86

verification-methodology

Guides agents through pass/fail verification using explicit criteria, direct source-faithful evidence, reproducible checks, and structured verdicts before declaring work complete.

magnus919/agent-skills ★ 88

verification testing pass-fail evidence

Pruebas 91

request-timing-audit

Audits iOS and macOS projects for permission requests made at launch or before meaningful user interaction, identifying likely App Store Guideline 5.1.1(ii) violations.