Skip to main content

​
Create a Test Suite

curl -X POST https://api.trainlyai.com/v1/{project_id}/testing/suites \
  -H "Authorization: Bearer tk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Translation Quality",
    "description": "Verify translation accuracy across languages.",
    "tags": ["translation", "regression"]
  }'
POST /v1/{project_id}/testing/suites

​
Request Body

ParameterTypeRequiredDescription
namestringYesSuite name
descriptionstringNoDescription of what this suite tests
tagslist[string]NoTags for organizing suites

​
Response

{
  "suite_id": "ste_a1b2c3d4",
  "name": "Translation Quality",
  "created_at": "2026-04-07T14:00:00Z"
}

​
List Test Suites

curl "https://api.trainlyai.com/v1/{project_id}/testing/suites" \
  -H "Authorization: Bearer tk_your_api_key"
GET /v1/{project_id}/testing/suites

​
Response

[
  {
    "suite_id": "ste_a1b2c3d4",
    "name": "Translation Quality",
    "description": "Verify translation accuracy across languages.",
    "tags": ["translation", "regression"],
    "case_count": 12,
    "created_at": "2026-04-07T14:00:00Z"
  }
]

​
Add a Test Case

curl -X POST https://api.trainlyai.com/v1/{project_id}/testing/suites/ste_a1b2c3d4/cases \
  -H "Authorization: Bearer tk_your_api_key" \
  -H "Content-Type: application/json" \
  -d '{
    "query": "Translate to French: Good morning",
    "expected_answer": "Bonjour",
    "category": "french",
    "assertions": [
      { "type": "contains", "value": "Bonjour" },
      { "type": "max_latency_ms", "value": 2000 }
    ]
  }'
POST /v1/{project_id}/testing/suites/{suite_id}/cases

​
Request Body

ParameterTypeRequiredDescription
querystringYesInput prompt for the test case
expected_answerstringYesGround-truth expected output
categorystringNoCategory label for grouping cases
assertionslist[object]NoAssertions to evaluate (see below)

​
Assertion Types

TypeValueDescription
containsstringOutput must contain the value
not_containsstringOutput must not contain the value
equalsstringOutput must exactly match the value
max_latency_msnumberResponse must complete within this time
min_scorenumberLLM judge score must meet this threshold

​
Response

{
  "case_id": "tsc_x9y8z7",
  "suite_id": "ste_a1b2c3d4",
  "created_at": "2026-04-07T14:05:00Z"
}

​
Run a Test Suite

curl -X POST https://api.trainlyai.com/v1/{project_id}/testing/suites/ste_a1b2c3d4/run \
  -H "Authorization: Bearer tk_your_api_key"
POST /v1/{project_id}/testing/suites/{suite_id}/run

​
Response

{
  "run_id": "run_m3n4o5p6",
  "suite_id": "ste_a1b2c3d4",
  "status": "running",
  "started_at": "2026-04-07T14:10:00Z"
}

​
Get Run Results

curl "https://api.trainlyai.com/v1/{project_id}/testing/runs/run_m3n4o5p6/results" \
  -H "Authorization: Bearer tk_your_api_key"
GET /v1/{project_id}/testing/runs/{run_id}/results

​
Response

{
  "run_id": "run_m3n4o5p6",
  "suite_id": "ste_a1b2c3d4",
  "status": "completed",
  "passed": 11,
  "failed": 1,
  "total": 12,
  "pass_rate": 0.9167,
  "results": [
    {
      "case_id": "tsc_x9y8z7",
      "query": "Translate to French: Good morning",
      "expected_answer": "Bonjour",
      "actual_output": "Bonjour",
      "passed": true,
      "latency_ms": 580.2,
      "assertions": [
        { "type": "contains", "value": "Bonjour", "passed": true },
        { "type": "max_latency_ms", "value": 2000, "passed": true }
      ]
    }
  ],
  "completed_at": "2026-04-07T14:10:47Z"
}