Core Concepts

Trainly’s observability model is built on five primitives.

Traces

A trace is a single recorded AI interaction — one input in, one output out. Every trace captures:

Field	Description
`input`	The prompt or user message
`output`	The AI response
`model`	Which LLM was used (e.g., `gpt-4o`)
`latency_ms`	Response time in milliseconds
`token_usage`	Prompt and completion token counts
`cost`	Estimated cost in USD
`tags`	Labels for filtering (e.g., `["production", "agent"]`)
`metadata`	Arbitrary key-value pairs
`status`	`success` or `error`

@client.observe(model="gpt-4o", tags=["production"])
def my_function(input):
    return llm_call(input)

Spans

A span is a sub-unit of work within a trace. Use spans to instrument multi-step pipelines — each step gets its own timing, attributes, and output.

with client.observe.span("retrieval", kind="retrieval") as s:
    docs = search(query)
    s.set_output(f"Found {len(docs)} documents")
    s.set_attribute("doc_count", len(docs))

Span kinds: chain, retrieval, tool, agent, llm, embedding.

Sessions

A session groups multiple traces into a single agent run. All traces within a session share a session_id for end-to-end correlation.

with client.observe.agent_session() as session_id:
    plan = plan_step(query)       # trace 1
    result = execute(plan)        # trace 2
    answer = synthesize(result)   # trace 3
# All 3 traces linked by session_id

Scores

Scores attach evaluation metrics to traces. Use them for human feedback, automated checks, or LLM-as-judge evaluation.

# Manual score (0-1 scale)
client.score(trace_id="tr_...", name="relevance", value=0.95)

# LLM judge score
client.score_with_judge(
    scorer_slug="helpfulness",
    trace_id="tr_...",
    input="What is X?",
    output="X is..."
)

Versions

Versions tag your pipeline configuration with semver strings. Publish a version, compare performance across versions, and rollback instantly.

client.versions.publish("1.2.0", description="Added retry logic")
# Later...
client.versions.rollback(version_id="ver_...")

Projects

A project is an isolated environment for traces. Each project has its own API key (tk_), project ID (proj_), and trace store. Use separate projects for different apps, environments, or teams.

​Core Concepts

​Traces

​Spans

​Sessions

​Scores

​Versions

​Projects