AI Eval in CI
Test AI agents and LLM outputs the same way you test code — automated evaluations that run in CI, compare against baselines, and fail the build when quality drops. No dashboards to check manually. Just npx eval run ci and a red or green build.
Overview
The AI Eval in CI skill, part of the TerminalSkills/skills repository (which has earned 71 stars), provides a framework for integrating Large Language Model (LLM) evaluation directly into continuous integration workflows. It allows developers to treat AI output testing with the same rigor as traditional software unit tests. By executing the command `npx eval run ci`, the tool performs automated evaluations and compares current results against established performance baselines. If the quality of the AI agent's response falls below the defined threshold, the CI build is triggered to fail, preventing regressions. This approach eliminates the need for manual dashboard monitoring by providing a binary pass/fail status within the developer's existing terminal-based or automated environment.
Use Cases
Install Notes
# Review source first
open https://github.com/TerminalSkills/skills/blob/main/skills/ai-eval-ci/SKILL.mdCopy or clone the skill folder into your agent skills directory after reviewing its instructions and scripts.
Security Notes
This skill operates within the user's CI environment and interacts with external LLM APIs as configured by the user. It requires appropriate permissions to execute shell commands and access the source repository. Users should ensure that API keys and sensitive data used during the evaluation process are managed securely through environment variables or CI secrets, as the tool facilitates automated testing across various AI agents including Claude and Gemini.
Related Skills
CI Cd And Automation
addyosmani/agent-skills
Automates CI/CD pipeline setup. Use when setting up or modifying build and deployment pipelines. Use when you need to automate quality gates, configure test runners in CI, or establish deployment strategies.
Observability And Instrumentation
addyosmani/agent-skills
Instruments code so production behavior is visible and diagnosable. Use when adding logging, metrics, tracing, or alerting. Use when shipping any feature that runs in production and you need evidence it works. Use when production issues are reported but you can't tell what happened from the available data.
Core
vercel-labs/agent-browser
Core agent-browser usage guide. Read this before running any agent-browser commands. Covers the snapshot-and-ref workflow, navigating pages, interacting with elements (click, fill, type, select), extracting text and data, taking screenshots, managing tabs, handling forms and auth, waiting for content, running multip...
Agentcore
vercel-labs/agent-browser
Run agent-browser on AWS Bedrock AgentCore cloud browsers. Use when the user wants to use AgentCore, run browser automation on AWS, use a cloud browser with AWS credentials, or needs a managed browser session backed by AWS infrastructure. Triggers include "use agentcore", "run on AWS", "cloud browser with AWS", "bed...
Agent Browser
vercel-labs/agent-browser
Browser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to "open a website", "fill out a form", "click a butto...
Netlify Forms
openai/plugins
Guide for using Netlify Forms for HTML form handling. Use when adding contact forms, feedback forms, file upload forms, or any form that should be collected by Netlify. Covers the data-netlify attribute, spam filtering, AJAX submissions, file uploads, notifications, and the submissions API.