Last modified July 27, 2026

TraceQL query reference

TraceQL is Grafana Tempo’s native query language for distributed trace analysis. Unlike log or metrics queries, TraceQL searches and filters across the hierarchical structure of traces and spans. This page is a reference for its syntax and the query patterns most useful on the Giant Swarm Observability Platform.

Tracing must be enabled on your cluster and your applications must be instrumented to send traces to otlp-gateway.kube-system.svc before these queries return data. See the data ingestion guide for setup, and the data exploration guide for accessing the Tempo data source in Grafana. For task-oriented troubleshooting with these queries, see troubleshoot traces with TraceQL. The authoritative language specification is the Grafana TraceQL documentation.

Trace data model

Traces consist of spans organized in a tree structure. Understanding this hierarchy is what makes TraceQL’s scope selectors meaningful:

Trace: The complete journey of a request through your system
Span: An individual operation within the trace (a service call, database query, etc.)
Root span: The entry point of the trace
Child spans: Operations triggered by parent spans

Syntax fundamentals

TraceQL queries follow this basic pattern:

{span.attribute = "value"}

Basic elements:

Attribute scopes: span., resource., event., link., instrumentation.. There’s no trace. attribute scope; trace-level fields are intrinsics.
Intrinsics: fields a span or trace always has, written with a colon. Common ones are span:duration, span:name, span:kind, span:status, trace:duration, and trace:rootService.
Attribute names: the specific attribute you want to filter on
Operators: =, !=, >, <, >=, <=, =~ (regex)
Values: quoted strings, numbers, durations (5s, 500ms), or unquoted keywords (error, client)

Essential queries

Finding traces by service

Query traces involving a specific service:

{resource.service.name = "user-service"}

Find traces involving multiple services:

{resource.service.name = "user-service"} && {resource.service.name = "payment-service"}

Filtering by operation name

Find specific operations within services:

{span:name = "GET /api/users"}

Use regex for pattern matching:

{span:name =~ "GET /api/.*"}

Performance-based filtering

Find slow traces (durations take units such as s, ms, or us):

{trace:duration > 5s}

Find traces with errors (the error status value is unquoted):

{span:status = error}

Combine conditions:

{trace:duration > 2s && span:status = error}

Advanced filtering techniques

HTTP-specific queries

Find traces for specific HTTP methods:

{span.http.method = "POST"}

Filter by HTTP status codes:

{span.http.status_code >= 400}

Find slow HTTP requests:

{span.http.method = "GET" && span:duration > 1s}

Database operation analysis

Find database queries:

{span.db.system = "postgresql"}

Analyze slow database operations:

{span.db.system = "postgresql" && span:duration > 500ms}

Find specific database operations:

{span.db.operation = "SELECT" && span.db.name = "users"}

Custom attribute filtering

Many applications add custom attributes to spans. Filter using these:

{span.custom.user_id = "12345"}

Find traces for specific customers or tenants:

{span.tenant.id = "customer-abc"}

Performance analysis patterns

Identifying bottlenecks

Find the slowest traces by setting a concrete duration threshold, then sort the result list by duration in Grafana:

{trace:duration > 10s}

Find services with high error rates:

{resource.service.name = "payment-service" && span:status = error}

Capacity planning queries

Find traces with more than one span from a service (aggregates take a trailing comparison):

{resource.service.name = "api-gateway"} | count() > 1

Analyze request patterns:

{span.http.method = "POST" && span.http.route = "/api/orders"}

Service graph exploration

Tempo generates service graphs from trace data. Use TraceQL to understand service interactions. For the service graph feature itself, see service graphs.

Group a service’s operations

Group a service’s spans by operation and keep the operations that occur more than once. A by() stage must be followed by an aggregate and a comparison:

{resource.service.name = "api-gateway"} | by(span:name) | count() > 1

Identify service communication patterns

Find outbound (client) calls, which represent one service calling another:

{span:kind = client}

Aggregation and structural queries

Aggregation functions

Keep only spansets with more than three matching spans:

{resource.service.name = "api-service"} | count() > 3

Average a numeric intrinsic or attribute across the matching spans:

{resource.service.name = "api-service"} | avg(span:duration) > 1s

Search aggregates are count(), avg(), max(), min(), and sum(). Percentiles aren’t a search aggregate. Compute them with a TraceQL metrics query instead.

Structural queries

Structural operators match spans by their position in the trace tree: > (direct child), >> (descendant), < (direct parent), << (ancestor), and ~ (sibling). There’s no parent. attribute prefix.

Find a database-query span that’s a direct child of a user-lookup span:

{span:name = "user-lookup"} > {span:name = "database-query"}

Query trace topology by root service:

{trace:rootService = "api-gateway" && span:kind = server}

Best practices

Query optimization

Start specific: Begin with service or operation names before adding duration filters
Use time ranges: Always specify time ranges to improve query performance
Limit results: Set a result limit in Grafana’s query editor for exploratory queries.

Common patterns

Error investigation:

{span:status = error} | by(resource.service.name) | count() > 1

Performance analysis:

{trace:duration > 5s} | by(resource.service.name) | avg(span:duration) > 1s

Span-kind breakdown:

{resource.service.name = "my-service"} | by(span:kind) | count() > 1

Avoiding common mistakes

Don’t over-filter: Too many conditions can return no results
Mind the hierarchy: Remember that traces have spans, not the other way around
Use appropriate time ranges: Excessively long time ranges can time out

Giant Swarm Offerings