Accueil / Skills / agent-evals-and-observability

agent-evals-and-observability

Provides framework- and vendor-neutral methods for evaluating AI agents, reviewing trajectories, analyzing regressions, setting release gates, and designing privacy-aware production observability.

★ 88 GitHub Stars

96 Score de qualité

24 Fichiers

About this Skill

Design, run, review, or release framework- and vendor-neutral evaluations and observability for AI agents. Use when defining agent evals, datasets, graders, trajectory review, regression analysis, release gates, production traces, or privacy-aware telemetry. Covers task and trajectory contracts, statistical comparisons, and incident-to-case learning; route framework implementation to pydanticai or langgraph when needed. Do not use this skill for unrelated requests; route to the nearest named specialist.

Cas d’usage

Define task and trajectory contracts
Build evaluation datasets and graders
Compare agent versions statistically
Review traces, tools, and side effects
Design production telemetry and release gates

Fonctionnalités

Dataset manifests and evaluation plans
Deterministic, human, model, and domain grading guidance
Trajectory and trace review methods
Statistical comparison and regression analysis guidance
Privacy-aware observability recommendations

Agents compatibles

Not specified

Skill files

Other


                                evals/evals.json

JSON Other


                                README.md

Markdown Other


                                templates/dataset-manifest.md

Markdown Other


                                templates/eval-plan.md

Markdown Other


                                templates/grader-specification.md

Markdown Other


                                templates/metric-mirror.md

Markdown Other


                                templates/release-gate.md

Markdown Other


                                templates/run-report.md

Markdown Other


                                templates/trace-review.md

Markdown Reference


                                references/datasets.md

Markdown Reference


                                references/evaluation-design.md

Markdown Reference


                                references/evidence-modes-and-metric-mirrors.md

Markdown Reference


                                references/framework-routing.md

Markdown Reference


                                references/graders.md

Markdown Reference


                                references/grounding-attribution.md

Markdown Reference


                                references/judge-stability.md

Markdown Reference


                                references/metrics-and-statistics.md

Markdown Reference


                                references/opentelemetry-genai.md

Markdown Reference


                                references/production-observability.md

Markdown Reference


                                references/regression-and-release.md

Markdown Reference


                                references/source-index.md

Markdown Reference


                                references/synthetic-scenarios.md

Markdown Reference


                                references/trajectory-review.md

Markdown SKILL.md


                                SKILL.md

Markdown

Related Skills

Tests 96

qa-methodology

Provides software teams with guidance, templates, references, and Python scripts for test strategy, regression, risk-based testing, CI triage, quality gates, exploratory testing, …

magnus919/agent-skills ★ 88

quality assurance test strategy test automation regression testing

Tests 96

playwright

Enables agents to author, run, debug, and triage Playwright end-to-end tests, integrate suites with CI, perform headless browsing and scraping, and check accessibility snapshots.

magnus919/agent-skills ★ 88

playwright e2e testing browser automation ci

Tests 86

verification-methodology

Guides agents through pass/fail verification using explicit criteria, direct source-faithful evidence, reproducible checks, and structured verdicts before declaring work complete.

magnus919/agent-skills ★ 88

verification testing pass-fail evidence

Tests 91

request-timing-audit

Audits iOS and macOS projects for permission requests made at launch or before meaningful user interaction, identifying likely App Store Guideline 5.1.1(ii) violations.