Skip to main content

​
Core Concepts

Trainly’s observability model is built on five primitives.

​
Traces

A trace is a single recorded AI interaction — one input in, one output out. Every trace captures:
FieldDescription
inputThe prompt or user message
outputThe AI response
modelWhich LLM was used (e.g., gpt-4o)
latency_msResponse time in milliseconds
token_usagePrompt and completion token counts
costEstimated cost in USD
tagsLabels for filtering (e.g., ["production", "agent"])
metadataArbitrary key-value pairs
statussuccess or error
@client.observe(model="gpt-4o", tags=["production"])
def my_function(input):
    return llm_call(input)

​
Spans

A span is a sub-unit of work within a trace. Use spans to instrument multi-step pipelines — each step gets its own timing, attributes, and output. 
with client.observe.span("retrieval", kind="retrieval") as s:
    docs = search(query)
    s.set_output(f"Found {len(docs)} documents")
    s.set_attribute("doc_count", len(docs))
Span kinds: chain, retrieval, tool, agent, llm, embedding.

​
Sessions

A session groups multiple traces into a single agent run. All traces within a session share a session_id for end-to-end correlation. 
with client.observe.agent_session() as session_id:
    plan = plan_step(query)       # trace 1
    result = execute(plan)        # trace 2
    answer = synthesize(result)   # trace 3
# All 3 traces linked by session_id

​
Scores

Scores attach evaluation metrics to traces. Use them for human feedback, automated checks, or LLM-as-judge evaluation. 
# Manual score (0-1 scale)
client.score(trace_id="tr_...", name="relevance", value=0.95)

# LLM judge score
client.score_with_judge(
    scorer_slug="helpfulness",
    trace_id="tr_...",
    input="What is X?",
    output="X is..."
)

​
Versions

Versions tag your pipeline configuration with semver strings. Publish a version, compare performance across versions, and rollback instantly. 
client.versions.publish("1.2.0", description="Added retry logic")
# Later...
client.versions.rollback(version_id="ver_...")

​
Projects

A project is an isolated environment for traces. Each project has its own API key (tk_), project ID (proj_), and trace store. Use separate projects for different apps, environments, or teams.