agent-evals-and-observability
Provides framework- and vendor-neutral methods for evaluating AI agents, reviewing trajectories, analyzing regressions, setting release gates, and designing privacy-aware producti…
Provides software teams with guidance, templates, references, and Python scripts for test strategy, regression, risk-based testing, CI triage, quality gates, exploratory testing, mutation testing, and agentic eval design.
Design and apply QA methodology for software teams: test strategy, regression testing, CI failure triage, test automation, quality gates and metrics, risk-based testing, exploratory testing, test design techniques, AI code quality gates (independent verification, artifact provenance, spec-first oracles, AI-review-comment triage, acceptance-criteria testability for Spec-Driven Development), mutation-guided test hardening and review evidence (surviving mutants, weak assertions, diff-aware mutation testing), agentic eval design (dataset test design, judge-as-system-under-test, flaky-eval discipline), QA career levels (Senior/Staff/Principal), and SDET engineering (test infrastructure, gTAA, CI/CD integration). Do not use for root-cause debugging of production incidents, security implementation or threat modeling, or evaluation framework governance and statistical analysis — route those to systematic-debugging, secure-software-engineering, and agent-evals-and-observability respectively.
assets/qa-definition-of-done.md
Markdown
Asset
assets/risk-matrix-grid.md
Markdown
Asset
assets/test-design-techniques-checklist.md
Markdown
Other
evals/evals.json
JSON
Other
pytest.ini
Other
README.md
Markdown
Other
templates/ai-assisted-verification-note.md
Markdown
Other
templates/bug-report.md
Markdown
Other
templates/exploratory-charter.md
Markdown
Other
templates/mutation-review.md
Markdown
Other
templates/risk-register.md
Markdown
Other
templates/test-strategy.md
Markdown
Other
templates/verification-plan.md
Markdown
Other
tests/test_check_ac_testability.py
Python
Other
tests/test_risk_prioritize.py
Python
Reference
references/agentic-eval-design.md
Markdown
Reference
references/ai-code-quality-gates.md
Markdown
Reference
references/ai-test-artifact-evidence.md
Markdown
Reference
references/ci-failure-triage.md
Markdown
Reference
references/exploratory-testing.md
Markdown
Reference
references/performance-testing.md
Markdown
Reference
references/qa-career-levels.md
Markdown
Reference
references/quality-gates-and-metrics.md
Markdown
Reference
references/regression-testing.md
Markdown
Reference
references/risk-based-testing.md
Markdown
Reference
references/sdet-engineering.md
Markdown
Reference
references/security-testing.md
Markdown
Reference
references/test-automation.md
Markdown
Reference
references/test-data-management.md
Markdown
Reference
references/test-debugging.md
Markdown
Reference
references/test-design-techniques.md
Markdown
Reference
references/test-strategy.md
Markdown
Script
scripts/check-ac-testability.py
Python
Script
scripts/risk-prioritize.py
Python
SKILL.md
SKILL.md
Markdown
Provides framework- and vendor-neutral methods for evaluating AI agents, reviewing trajectories, analyzing regressions, setting release gates, and designing privacy-aware producti…
Enables agents to author, run, debug, and triage Playwright end-to-end tests, integrate suites with CI, perform headless browsing and scraping, and check accessibility snapshots.
Guides agents through pass/fail verification using explicit criteria, direct source-faithful evidence, reproducible checks, and structured verdicts before declaring work complete.
Audits iOS and macOS projects for permission requests made at launch or before meaningful user interaction, identifying likely App Store Guideline 5.1.1(ii) violations.
Audits iOS and macOS projects for misuse of background execution modes, including silent audio, location, VoIP, and unregistered background tasks under Apple Guideline 2.5.4.
Checks iOS and macOS projects for missing or incomplete PrivacyInfo.xcprivacy files and required-reason API declarations before App Store submission.
Audits iOS and macOS projects for missing or noncompliant Sign in with Apple implementation when third-party authentication is present, using static file and dependency searches.
Audits iOS source code for fixed fonts, fixed-height text containers, and other patterns that prevent text from scaling with Dynamic Type.