Testing Platform

Testing Platform

Ship AI agents with confidence. Keep a catalog of test cases, report results from your existing test runner (Playwright, Vitest, pytest, and anything else that speaks CTRF), and track runs over time with full deployment traceability — from the CLI or your CI/CD pipeline.

Core Concepts

Test Cases

Catalog individual test scenarios with a title, description, expected result, priority, tags, and automation status.

Test Environments

Group runs by where they executed (development, staging, production). Environments are created on demand when you report by name.

Test Runs

A run records the outcome of a test execution — pass/fail/skip summary, duration, tool, and per-test results ingested from CTRF.

Deployments

Track agent deployments across environments. Link runs to a specific deployment version for traceability.

CLI in Action

The th CLI — a single Rust binary — manages and reports test runs directly from your terminal or CI/CD pipeline. Testing commands live under smoo api testing.

smoo api testing
$ smoo auth login --m2m
Authenticated as brent@smoo.ai
$ smoo api testing cases list
ID Title Automation
case_abc123 Greeting Test automated
case_def456 Order Lookup automated
case_ghi789 Escalation Flow automated
case_jkl012 Edge Case: Empty Input manual
$ smoo api testing runs report ./ctrf-report.json --environment staging --deployment-id v2.1.0
Uploading CTRF report... done
Run ID: run_abc123
Parsed 4 results from ctrf-report.json
✓ Greeting Test passed (1.2s)
✓ Order Lookup passed (2.1s)
✓ Escalation Flow passed (1.8s)
✗ Edge Case: Empty Input failed (0.4s)
Results: 3/4 passed, 1 failed
curl -fsSL https://raw.githubusercontent.com/SmooAI/smooth/main/install.sh | sh

Multi-Language Support: The th CLI is a single static binary that works from any language or environment. The @smooai/testing SDK is currently TypeScript-only. For programmatic access from other languages, use the REST API to report from any language or framework that can produce CTRF.

CTRF Integration

Smoo Testing natively supports the Common Test Report Format (CTRF). Results from any CTRF-compatible runner flow into the Smoo AI dashboard — reporting a CTRF file creates a run and parses each test result.

import { SmooTestingClient } from '@smooai/testing';

const client = new SmooTestingClient({
    clientId: process.env.SMOOAI_CLIENT_ID!,
    clientSecret: process.env.SMOOAI_CLIENT_SECRET!,
    orgId: process.env.SMOO_ORG_ID!,
});

// Report a CTRF file from any test runner — creates a run and parses results
const run = await client.report('./ctrf-report.json', {
    name: 'PR #42 Tests',
    environment: 'staging',
    tool: 'playwright',
});
console.log(run.summary);
// { total: 4, passed: 3, failed: 1, skipped: 0 }

Supported Test Runners

Any runner with a CTRF reporter: Playwright, Jest, Vitest, Mocha, pytest, JUnit, and more. Results are automatically parsed and displayed in the Smoo AI dashboard.

GitHub Integration

Connect your GitHub repository to sync deployments and environments and trigger workflows. Deployments created from GitHub give your test runs a version to link back to. See the Integrations guide for full setup.

Deployment Sync

GitHub deployment events are synced into Smoo AI deployment records with commit metadata.

Environment Tracking

GitHub environments are synced as Smoo AI test environments so runs group consistently.

Workflow Dispatch

Trigger GitHub Actions workflows from the Smoo AI dashboard to run and report your tests.

SmooTestingClient SDK

The TypeScript SDK provides programmatic access to the testing API. Manage test cases, create and report runs, and fetch results from your application code. Authenticate with an M2M client-credentials pair (see Authentication).

import { SmooTestingClient } from '@smooai/testing';

const testing = new SmooTestingClient({
    clientId: process.env.SMOOAI_CLIENT_ID!,
    clientSecret: process.env.SMOOAI_CLIENT_SECRET!,
    orgId: process.env.SMOO_ORG_ID!,
});

// Catalog a test case
const testCase = await testing.createCase({
    title: 'Order Status Inquiry',
    description: 'Agent returns order status and estimated delivery date',
    expectedResult: 'Response includes order status and a delivery estimate',
    priority: 'medium',
    automationStatus: 'automated',
    tags: ['orders'],
});

// Create a run, then fetch its results
const run = await testing.createRun({
    name: 'Nightly',
    tool: 'playwright',
    environment: 'staging',
});
const results = await testing.getRun(run.id);
console.log(results.summary);
// { total: 1, passed: 1, failed: 0 }

CI/CD Pipeline

Report tests automatically on every deployment. Here's a GitHub Actions workflow that runs your test suite after deploying to staging, then reports the CTRF results to Smoo AI with the th CLI.

name: Test AI Agent
on:
  push:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Deploy to staging
        run: pnpm deploy:staging

      - name: Install th CLI
        run: curl -fsSL https://raw.githubusercontent.com/SmooAI/smooth/main/install.sh | sh

      - name: Run tests & report to Smoo AI
        env:
          SMOOAI_CLIENT_ID: ${{ secrets.SMOOAI_CLIENT_ID }}
          SMOOAI_CLIENT_SECRET: ${{ secrets.SMOOAI_CLIENT_SECRET }}
        run: |
          smoo auth login --m2m
          smoo api testing runs report ./ctrf-report.json \
            --environment staging \
            --deployment-id ${{ github.sha }}

smoo api testing runs report Flags

  • --environment — Associate the run with a test environment (created on demand)
  • --deployment-id — Link the run to a specific deployment
  • --name — Set a human-readable name for the run
  • --tool — Record which test runner produced the report

REST API Examples

Create a Test Case

const res = await fetch("https://api.smoo.ai/organizations/{org_id}/testing/cases", {
    method: "POST",
    headers: {
        Authorization: `Bearer ${process.env.SMOO_ACCESS_TOKEN}`,
        "Content-Type": "application/json",
    },
    body: JSON.stringify({
        "title": "Greeting Test",
        "description": "Verify the agent responds with a friendly greeting",
        "expectedResult": "Response contains a greeting and asks for order details",
        "priority": "medium",
        "automationStatus": "automated",
        "tags": [
            "smoke"
        ]
    }),
});
const data = await res.json();
console.log(data);

Create a Test Environment

const res = await fetch("https://api.smoo.ai/organizations/{org_id}/testing/environments", {
    method: "POST",
    headers: {
        Authorization: `Bearer ${process.env.SMOO_ACCESS_TOKEN}`,
        "Content-Type": "application/json",
    },
    body: JSON.stringify({
        "name": "Staging",
        "description": "Pre-production testing environment"
    }),
});
const data = await res.json();
console.log(data);

Create a Test Run

const res = await fetch("https://api.smoo.ai/organizations/{org_id}/testing/runs", {
    method: "POST",
    headers: {
        Authorization: `Bearer ${process.env.SMOO_ACCESS_TOKEN}`,
        "Content-Type": "application/json",
    },
    body: JSON.stringify({
        "name": "PR #42 Tests",
        "tool": "playwright",
        "environment": "staging"
    }),
});
const data = await res.json();
console.log(data);

Submit Results (CTRF)

const res = await fetch("https://api.smoo.ai/organizations/{org_id}/testing/runs/{run_id}/results", {
    method: "POST",
    headers: {
        Authorization: `Bearer ${process.env.SMOO_ACCESS_TOKEN}`,
        "Content-Type": "application/json",
    },
    body: JSON.stringify({
        "results": {
            "tool": {
                "name": "playwright"
            },
            "summary": {
                "tests": 4,
                "passed": 3,
                "failed": 1
            },
            "tests": [
                {
                    "name": "Greeting Test",
                    "status": "passed",
                    "duration": 1200
                }
            ]
        }
    }),
});
const data = await res.json();
console.log(data);

Get Test Run Results

const res = await fetch("https://api.smoo.ai/organizations/{org_id}/testing/runs/{run_id}", {
    method: "GET",
    headers: {
        Authorization: `Bearer ${process.env.SMOO_ACCESS_TOKEN}`,
    },
});
const data = await res.json();
console.log(data);

Typical Workflow

  1. 1

    Catalog test cases

    Record the test cases that cover your agent's expected behavior: greetings, FAQs, edge cases, and escalation scenarios.

  2. 2

    Set up environments

    Create test environments that mirror your deployment stages (development, staging, production).

  3. 3

    Report results on each deployment

    Run your suite after each deployment and report the CTRF results to catch regressions. Use the CLI or GitHub Actions for automation.

  4. 4

    Review results and iterate

    Analyze run results in the dashboard, identify failing cases, and refine your agent's knowledge base and prompts.