1. Docs
  2. Overview
  3. Observability
  4. Data management
  5. Data transformation
  6. Alert on trace-derived metrics

Last modified July 8, 2026

Alert on trace-derived metrics

This guide shows you how to build alert rules from trace-derived metrics. Because these are ordinary Prometheus metrics, you alert on them exactly as you would any other metric — see the alert rules guide for how rules are defined and deployed on the platform. The examples below are the trace-specific expressions to drop into that workflow.

Alert on a high service error rate

Alert when a service’s error rate (from service graph metrics) exceeds a threshold:

groups:
- name: trace-based-alerts
  rules:
  - alert: HighServiceErrorRate
    expr: |
      (
        sum(rate(tempo_service_graph_request_failed_total[5m])) by (server) /
        sum(rate(tempo_service_graph_request_total[5m])) by (server)
      ) * 100 > 5      
    for: 5m
    labels:
      severity: warning
    annotations:
      summary: "High error rate detected for service {{ $labels.server }}"
      description: "Service {{ $labels.server }} has error rate of {{ $value }}% for 5 minutes"

Alert on high service latency

Alert when the 95th percentile latency for a service crosses a threshold:

- alert: HighServiceLatency
  expr: |
    histogram_quantile(0.95,
      sum(rate(tempo_service_graph_request_duration_seconds_bucket[5m])) by (server, le)
    ) > 2    
  for: 10m
  labels:
    severity: critical
  annotations:
    summary: "High latency detected for service {{ $labels.server }}"
    description: "Service {{ $labels.server }} 95th percentile latency is {{ $value }}s"

Alert on an unavailable service

Alert when a service stops producing request metrics, which suggests it’s unavailable:

- alert: ServiceUnavailable
  expr: |
    absent_over_time(
      sum(rate(tempo_service_graph_request_total[1m])) by (server)[5m:]
    ) == 1    
  labels:
    severity: critical
  annotations:
    summary: "Service {{ $labels.server }} appears to be unavailable"
    description: "No requests detected for service {{ $labels.server }} in the last 5 minutes"

Alert design principles

When designing alerts on trace-derived metrics, keep them actionable:

  • Focus on business impact: alert on conditions that affect user experience, not on every fluctuation.
  • Use appropriate time windows: balance sensitivity against noise with a sensible for duration.
  • Set meaningful thresholds: base thresholds on historical data and your SLA requirements rather than round numbers.
  • Include context: add labels and annotations that help responders act quickly.

See also