Create a Test Suite
curl -X POST https://api.trainlyai.com/v1/{project_id}/testing/suites \
-H "Authorization: Bearer tk_your_api_key" \
-H "Content-Type: application/json" \
-d '{
"name": "Translation Quality",
"description": "Verify translation accuracy across languages.",
"tags": ["translation", "regression"]
}'
POST /v1/{project_id}/testing/suites
Request Body
| Parameter | Type | Required | Description |
|---|---|---|---|
name | string | Yes | Suite name |
description | string | No | Description of what this suite tests |
tags | list[string] | No | Tags for organizing suites |
Response
{
"suite_id": "ste_a1b2c3d4",
"name": "Translation Quality",
"created_at": "2026-04-07T14:00:00Z"
}
List Test Suites
curl "https://api.trainlyai.com/v1/{project_id}/testing/suites" \
-H "Authorization: Bearer tk_your_api_key"
GET /v1/{project_id}/testing/suites
Response
[
{
"suite_id": "ste_a1b2c3d4",
"name": "Translation Quality",
"description": "Verify translation accuracy across languages.",
"tags": ["translation", "regression"],
"case_count": 12,
"created_at": "2026-04-07T14:00:00Z"
}
]
Add a Test Case
curl -X POST https://api.trainlyai.com/v1/{project_id}/testing/suites/ste_a1b2c3d4/cases \
-H "Authorization: Bearer tk_your_api_key" \
-H "Content-Type: application/json" \
-d '{
"query": "Translate to French: Good morning",
"expected_answer": "Bonjour",
"category": "french",
"assertions": [
{ "type": "contains", "value": "Bonjour" },
{ "type": "max_latency_ms", "value": 2000 }
]
}'
POST /v1/{project_id}/testing/suites/{suite_id}/cases
Request Body
| Parameter | Type | Required | Description |
|---|---|---|---|
query | string | Yes | Input prompt for the test case |
expected_answer | string | Yes | Ground-truth expected output |
category | string | No | Category label for grouping cases |
assertions | list[object] | No | Assertions to evaluate (see below) |
Assertion Types
| Type | Value | Description |
|---|---|---|
contains | string | Output must contain the value |
not_contains | string | Output must not contain the value |
equals | string | Output must exactly match the value |
max_latency_ms | number | Response must complete within this time |
min_score | number | LLM judge score must meet this threshold |
Response
{
"case_id": "tsc_x9y8z7",
"suite_id": "ste_a1b2c3d4",
"created_at": "2026-04-07T14:05:00Z"
}
Run a Test Suite
curl -X POST https://api.trainlyai.com/v1/{project_id}/testing/suites/ste_a1b2c3d4/run \
-H "Authorization: Bearer tk_your_api_key"
POST /v1/{project_id}/testing/suites/{suite_id}/run
Response
{
"run_id": "run_m3n4o5p6",
"suite_id": "ste_a1b2c3d4",
"status": "running",
"started_at": "2026-04-07T14:10:00Z"
}
Get Run Results
curl "https://api.trainlyai.com/v1/{project_id}/testing/runs/run_m3n4o5p6/results" \
-H "Authorization: Bearer tk_your_api_key"
GET /v1/{project_id}/testing/runs/{run_id}/results
Response
{
"run_id": "run_m3n4o5p6",
"suite_id": "ste_a1b2c3d4",
"status": "completed",
"passed": 11,
"failed": 1,
"total": 12,
"pass_rate": 0.9167,
"results": [
{
"case_id": "tsc_x9y8z7",
"query": "Translate to French: Good morning",
"expected_answer": "Bonjour",
"actual_output": "Bonjour",
"passed": true,
"latency_ms": 580.2,
"assertions": [
{ "type": "contains", "value": "Bonjour", "passed": true },
{ "type": "max_latency_ms", "value": 2000, "passed": true }
]
}
],
"completed_at": "2026-04-07T14:10:47Z"
}