1. Docs
  2. Overview
  3. Observability
  4. Data management
  5. Data exploration
  6. TraceQL query reference

Last modified July 7, 2026

TraceQL query reference

TraceQL is Grafana Tempo’s native query language for distributed trace analysis. Unlike log or metrics queries, TraceQL searches and filters across the hierarchical structure of traces and spans. This page is a reference for its syntax and the query patterns most useful on the Giant Swarm Observability Platform.

Tracing must be enabled on your cluster and your applications must be instrumented to send traces to otlp-gateway.kube-system.svc before these queries return data. See the data ingestion guide for setup, and the data exploration guide for accessing the Tempo data source in Grafana. For task-oriented troubleshooting with these queries, see troubleshoot traces with TraceQL. The authoritative language specification is the Grafana TraceQL documentation.

Trace data model

Traces consist of spans organized in a tree structure. Understanding this hierarchy is what makes TraceQL’s scope selectors meaningful:

  • Trace: The complete journey of a request through your system
  • Span: An individual operation within the trace (a service call, database query, etc.)
  • Root span: The entry point of the trace
  • Child spans: Operations triggered by parent spans

Syntax fundamentals

TraceQL queries follow this basic pattern:

{span.attribute = "value"}

Basic elements:

  • Scope selectors: span., resource., trace.
  • Attribute names: The specific attribute you want to filter on
  • Operators: =, !=, >, <, >=, <=, =~ (regex)
  • Values: Strings, numbers, or durations

Essential queries

Finding traces by service

Query traces involving a specific service:

{resource.service.name = "user-service"}

Find traces involving multiple services:

{resource.service.name = "user-service"} && {resource.service.name = "payment-service"}

Filtering by operation name

Find specific operations within services:

{span.name = "GET /api/users"}

Use regex for pattern matching:

{span.name =~ "GET /api/.*"}

Performance-based filtering

Find slow traces (duration in nanoseconds):

{trace.duration > 5s}

Find traces with errors:

{status = error}

Combine conditions:

{trace.duration > 2s && status = error}

Advanced filtering techniques

HTTP-specific queries

Find traces for specific HTTP methods:

{span.http.method = "POST"}

Filter by HTTP status codes:

{span.http.status_code >= 400}

Find slow HTTP requests:

{span.http.method = "GET" && span.duration > 1s}

Database operation analysis

Find database queries:

{span.db.system = "postgresql"}

Analyze slow database operations:

{span.db.system = "postgresql" && span.duration > 500ms}

Find specific database operations:

{span.db.operation = "SELECT" && span.db.name = "users"}

Custom attribute filtering

Many applications add custom attributes to spans. Filter using these:

{span.custom.user_id = "12345"}

Find traces for specific customers or tenants:

{span.tenant.id = "customer-abc"}

Performance analysis patterns

Identifying bottlenecks

Find the slowest traces by setting a concrete duration threshold, then sort the result list by duration in Grafana:

{trace.duration > 10s}

Find services with high error rates:

{resource.service.name = "payment-service" && status = error}

Capacity planning queries

Count spans per service:

{resource.service.name = "api-gateway"} | count()

Analyze request patterns:

{span.http.method = "POST" && span.http.route = "/api/orders"}

Service graph exploration

Tempo generates service graphs from trace data. Use TraceQL to understand service interactions. For the service graph feature itself, see service graphs.

Analyze service dependencies

Find all services called by a specific service:

{resource.service.name = "api-gateway"} | by(resource.service.name)

Identify service communication patterns

Find cross-service calls:

{span.kind = "client"} && {resource.service.name != parent.resource.service.name}

Aggregation and structural queries

Aggregation functions

Count traces matching criteria:

{resource.service.name = "api-service"} | count()

Calculate percentiles:

{resource.service.name = "api-service"} | quantile(0.95)

Structural queries

Find traces with specific span relationships:

{span.name = "database-query" && parent.span.name = "user-lookup"}

Query trace topology:

{trace.root.service.name = "api-gateway" && span.kind = "server"}

Best practices

Query optimization

  • Start specific: Begin with service or operation names before adding duration filters
  • Use time ranges: Always specify time ranges to improve query performance
  • Limit results: Use | limit(100) for exploratory queries

Common patterns

  1. Error investigation:

    {status = error} | by(resource.service.name) | count()

  2. Performance analysis:

    {trace.duration > 5s} | by(resource.service.name) | quantile(0.95)

  3. Service dependency mapping:

    {resource.service.name = "my-service"} | by(span.kind, resource.service.name)

Avoiding common mistakes

  • Don’t over-filter: Too many conditions can return no results
  • Mind the hierarchy: Remember that traces have spans, not the other way around
  • Use appropriate time ranges: Excessively long time ranges can time out

See also