This is documentation for the next version of Grafana documentation. For the latest stable release, go to the latest version.

Documentation Grafana documentation Data sources Tempo

Tempo data source

Grafana ships preinstalled with the Tempo data source for Grafana Tempo, a high-volume, minimal-dependency distributed tracing backend from Grafana Labs. There’s nothing to install to get started. Use the Tempo data source to search and visualize traces, correlate traces with logs, metrics, and profiles, and monitor service dependencies with the Service Graph.

As of Grafana 13.2, Tempo is packaged as a standalone plugin so it can receive updates independently of Grafana releases. For details, refer to Plugin updates.

Want to learn more about traces and the other telemetry signals? Refer to Correlate telemetry signals.

Supported features

The Tempo data source supports the following features:

Get started

The following pages help you set up and use the Tempo data source:

• Configure the Tempo data source: Connect Grafana to Tempo, set up authentication, and configure trace correlations.
• Query tracing data: Search for traces, use the TraceQL editor, and upload JSON trace files.
• Grafana Traces Drilldown: Explore tracing data visually using RED metrics, without writing queries.
• Service Graph and Service Graph view: Visualize service dependencies and monitor request rate, error rate, and duration. Requires a linked Prometheus data source with service graph metrics.

To explore tracing data from a Tempo data source, try Grafana Traces Drilldown on Grafana Play:

Give it a try using Grafana Play

With Grafana Play, you can explore and see how it works, learning from practical examples to accelerate your development. This feature can be seen on Grafana Traces Drilldown.

Try it

Connect traces to other signals

After you’ve connected Grafana to Tempo, you can configure correlations between traces and other signals:

• Trace to logs: Navigate from spans to related logs in Loki, including bidirectional linking.
• Trace to metrics: Link spans to metrics queries in Prometheus or other metrics data sources.
• Trace to profiles: Link spans to profiling data in Grafana Pyroscope with embedded flame graphs.
• Trace correlations: Create custom correlation links to any data source or external URL.

Plugin updates

Starting with Grafana v13.2, the Tempo data source is a standalone plugin, preinstalled in both Grafana OSS and Enterprise. This enables more frequent updates independent of Grafana releases. Grafana automatically checks the plugin catalog on each server restart and installs the latest version within the current major version. New major versions aren’t installed automatically.

To adjust this behavior:

• Opt out of auto-updates: Set preinstall_auto_update to false in your configuration file.
• Update manually: Update at any time from the Administration > Plugins page without restarting Grafana.

The standalone plugin requires Grafana 12.3.0 or later. The Tempo data source bundled with Grafana 13.1 and earlier continues to work as before. These versions are unaffected by the externalization.

Users running Grafana 12.3.x through 13.1.x can install the standalone plugin from the plugin catalog if they want the latest features before upgrading to Grafana 13.2. To use the standalone plugin with these versions, add the following to your configuration file:

ini Copy

[plugin.tempo]
as_external = true

[plugins]
; Install the latest version on startup:
preinstall_sync = tempo
; Or install a specific version:
; preinstall_sync = tempo@<version>

For example, to search and query your Tempo data source, use the gcx datasources tempo commands:

sh Copy

# Search for traces with a TraceQL query
gcx datasources tempo query -d <DATASOURCE_UID> '{ span.http.status_code >= 500 }' --since 1h

# Retrieve a single trace by its ID
gcx datasources tempo get -d <DATASOURCE_UID> <TRACE_ID>

# List all trace labels, or the values for a single label
gcx datasources tempo labels -d <DATASOURCE_UID>
gcx datasources tempo labels -d <DATASOURCE_UID> --label service.name

# Run a TraceQL metrics query that returns time series
gcx datasources tempo metrics -d <DATASOURCE_UID> '{ } | rate()' --since 1h

Replace <DATASOURCE_UID> with the UID of your Tempo data source and <TRACE_ID> with a hex-encoded trace ID. You can omit the -d flag when datasources.tempo is configured in your gcx context. The query command searches for traces, get retrieves a single trace by ID, and metrics returns time series from a TraceQL metrics query.

Related resources

• Introduction to tracing
• Best practices for traces: Guidance on planning spans, attributes, and trace structure for effective tracing data.
• TraceQL query examples
• TraceQL query language reference If you encounter issues with the Tempo data source, refer to Troubleshoot Tempo data source issues.

• Configure the Tempo data source
  Guide for configuring Tempo in Grafana
• Query tracing data
  Guide for using the Tempo data source's query editor
• Service Graph and Service Graph view
  Use the Service Graph and Service Graph view to visualize service dependencies and monitor RED metrics
• Span filters
  Use span filters to filter spans in the timeline viewer
• Troubleshoot Tempo data source issues
  Troubleshoot common problems with the Tempo data source in Grafana