Documentation Grafana Cloud Observe and act Connect to data sources Data sources Tempo Query tracing data

Query tracing data

The Tempo data source’s query editor helps you query and display traces from Tempo in Explore. The queries use TraceQL, the query language designed specifically for tracing.

For general documentation on querying data sources in Grafana, refer to Query and transform data.

Before you begin

You can compose TraceQL queries in Grafana and Grafana Cloud using Explore and a Tempo data source.

You don’t have to know TraceQL to create a query. The Search query builder lets you select options from drop-down lists to build a query visually. These selections generate a TraceQL query that you can transfer to the TraceQL editor to refine further.

TraceQL queries follow the pattern { conditions } | pipeline, where conditions inside {} select spans and pipeline operators like count() or avg() aggregate results. To learn the full syntax, refer to Construct a TraceQL query. For a language overview, refer to the TraceQL documentation.

Choose a query editing mode

The query editor has four Query types that you can use to explore your tracing data. You can use these modes by themselves or in combination to create building blocks to generate custom queries. Adding another query adds a new query block. Refer to Use query types together for more information.

Search query builder

Start here if you’re exploring data or learning TraceQL. The Search query builder provides drop-down lists and text fields to build a query visually without needing to know TraceQL syntax. Your selections automatically generate a TraceQL query that you can copy to the editor for further refinement.

Refer to Search traces using the query builder for more information.

TraceQL query editor

Use the TraceQL editor when you need complex filters, structural queries across parent and child spans, or aggregations that the Search builder doesn’t support. The editor provides autocomplete for attribute names, scopes, and operators. To look up a single trace by its ID, use the Trace ID query type.

For copy-paste query examples, refer to TraceQL query examples. Refer to Write TraceQL queries with the editor for more information.

Trace ID query

Use the Trace ID query type to retrieve a single trace when you already know its ID, for example from a log line, an error report, or a link. Unlike the TraceQL editor, it provides options to shrink the returned trace: you can collapse repeated spans with span pruning and filter the trace to the spans you care about.

To look up a trace:

1. Select the Trace ID query type.
2. Enter the trace ID in the Trace ID field. The field is marked invalid if the value doesn’t start with letters and numbers.
3. Press Enter to run the query.

The Trace ID field only accepts a trace ID.

Span pruning options

Span pruning collapses groups of similar leaf spans into a single summary span, which makes large traces easier to read and smaller to load. Each summary span records how many spans it replaces.

Expand Span Pruning Options to change how pruning behaves. The collapsed section shows a summary of the current values.

On/Off

Turns span pruning on or off for this query. Default value is On.

Group By

A comma-separated list of attribute names used to group spans for pruning, for example db.sql.table, http.method. Spans are grouped when they share a name, kind, and status. Spans must also have identical values for every attribute listed here. By default, no attributes are used.

Min Spans

The minimum number of similar spans required in a group before pruning is applied. Groups smaller than this value aren’t pruned. Default value is 5.

Max Parent Depth

How many ancestor levels above the aggregated leaf spans can also be aggregated. Use 0 to aggregate only leaves, -1 for unlimited depth, or n for a bounded depth. Default value is 1.

Filter options

Filter options reduce the returned trace to the spans that match a TraceQL filter. Expand Filter Options to set them. You can expand Span Pruning Options and Filter Options independently.

Filter

A TraceQL spanset filter evaluated against the spans in the trace, for example { status = error && name = "foo" }. The filter supports a single { ... } block. Chained filters and pipelines aren’t supported. Leave the field empty to skip filtering. The field shows an error when the filter isn’t valid.

Keep Hierarchy

Includes each matched span’s ancestor path back to the root, so the result displays as a complete waterfall instead of only the matching spans. Default value is Off.

Match Depth

How many levels of descendants to keep below each matched span, independent of Keep Hierarchy. Use -1 to keep all descendants, 0 for only the matched spans, or n for a bounded depth. Default value is 0.

Ancestor Depth

How many levels of ancestors to keep above each matched span. Use -1 for the whole path to the root, 0 for none, or n for a bounded depth. Only used when Keep Hierarchy is on. Default value is -1.

Keep Hierarchy, Match Depth, and Ancestor Depth are disabled until Filter contains a valid filter. Ancestor Depth is also disabled while Keep Hierarchy is off. Select Clear Filter to remove the filter and reset all filter options.

Example: Focus on one service in a large trace

A trace that spans many services can contain thousands of spans, and most of them are rarely relevant to the problem you’re investigating. Use Filter and Match Depth to load only the part of the trace you care about.

For example, you’re investigating slow requests in the checkout service, and the trace passes through dozens of other services. To show only the work that starts in checkout:

1. In Filter Options, set Filter to { resource.service.name = "checkout" }.
2. Set Match Depth to -1.
3. Run the query.

The filter matches every span that belongs to the checkout service. Because Match Depth is -1, Tempo also keeps all descendants of each matched span, so you still see everything those spans called downstream. The rest of the trace is left out.

To see where checkout sits in the wider request, turn Keep Hierarchy on. Tempo then also keeps each matched span’s ancestors back to the root, so the result displays as a complete waterfall. Use Ancestor Depth to limit how many ancestor levels are kept.

To narrow the result further, add conditions to the filter, for example { resource.service.name = "checkout" && status = error }. You can combine the filter with span pruning to also collapse repeated spans in the part of the trace you kept.

Service graph view

The Service Graph view answers a different question than trace search: instead of finding individual traces, it visualizes how your services connect and highlights request rate, error rate, and duration (RED metrics) across those connections.

Using the Service Graph view, you can:

• Discover spans that consistently return errors and the rates at which they occur.
• Get an overview of the overall rate of span calls throughout your services.
• Determine how long the slowest queries in your service take to complete.
• Examine all traces that contain spans of particular interest based on rate, error, and duration values (RED signals).

The Service Graph requires metrics generation to be configured. For more information, refer to Service Graph and Service Graph view.

Understand search behavior

Tempo search is non-deterministic. When you run a search, Tempo performs a massively parallel scan over the selected time range and returns the first N matching results. Even identical searches can return different results due to variations in machine load and network latency.

This design prioritizes speed over predictability. To make results deterministic, append a with clause to your TraceQL queries, for example { status = error } with (most_recent=true), so Tempo returns the most recent matching traces. For more information, refer to Retrieve most recent results.

Use TraceQL panels in dashboards

To add TraceQL panels to your dashboard, refer to the Traces panel documentation.

To learn more about Grafana dashboards, refer to the Use dashboards documentation.

For example dashboards with traces, visit Grafana Play.

Set options for query builder and editor

The following options are available for the Search and TraceQL query types. You can modify these settings in the Options section. The Trace ID query type has its own options, described in Trace ID query.

After changing any option, re-run the query to apply the updates.

Limit

Determines the maximum number of traces to return. Default value is 20.

Span Limit

Sets the maximum number of spans to return for each spanset. Default value is 3.

Table Format

Determines whether the query results table is displayed focused on Traces or Spans. Traces is the default selection. When Traces is selected, the results table starts with the trace ID. When Spans is selected, the table starts with the trace service.

Step

Defines the step for metrics queries. Use duration notation, for example, 30ms or 1m.

Streaming

Indicates if streaming is active. Streaming lets you view partial query results before the entire query completes. Activating streaming adds the Table - Streaming Progress section to the query results.

Use query types together

You can use + Add query to create customized queries that use one or more of the query types together. Each time you add a new query, it adds a new section, or query block, that contains Search, TraceQL, Trace ID, or Service Graph user interface.

The added query and results table appear in the navigation under Queries and Tables respectively. You can use the navigation to view query, results table, and service graph blocks.

To add a query block:

1. Select + Add query.
2. Choose a query type: Search, TraceQL, Trace ID, or Service Graph.

To remove a query block, select the Remove query trash can icon.

To rename a block, select the Rename edit icon next to the query block name. The name changes in the queries and table list.

Additional query block options

Each query block has a set of icons in the right top corner.

These icons include:

Show data source help

Displays the Tempo Cheat Sheet with links to documentation.

Create recorded query

Lets you save the current query block as a recorded query. This option is available in Grafana Cloud and Grafana Enterprise. For more information, refer to Recorded queries.

Duplicate query

Copies the current block and adds a new identical block.

Remove query

Deletes the query block.

Use query history and query inspector

Explore provides a history of all queries you’ve used within a data source and an inspector that lets you view stats, inspect queries, view JSON, and general information for your data source queries.

For more information, refer to the Query inspector in Explore and Query management in Explore documentation.

Upload a JSON trace file

You can upload a JSON file that contains a single trace and visualize it. If the file has multiple traces, Grafana visualizes the first trace.

To upload a trace file:

1. Select Explore in Grafana.
2. Select Import trace next to the query type selector.
3. Upload your JSON trace file.

To download a trace or Service Graph through the Inspector panel:

1. Open the inspector.
2. Navigate to the Data tab.
3. Click Download traces or Download Service Graph.

Cross-tenant TraceQL queries

If you’ve configured a multi-stack Tempo data source, you can perform TraceQL queries across those stacks and tenants.

Queries performed using the cross-tenant configured data source, in either Explore or inside of dashboards, are performed across all the tenants that you specified in the X-Scope-OrgID header.

TraceQL queries that compare multiple spansets may not correctly return all traces in a cross-tenant query. For instance,

Copy

{ span.attr1 = "bar" } && { span.attr2 = "foo" }

TraceQL evaluates a contiguously stored trace. If these two conditions are satisfied in separate tenants, then Tempo doesn’t return the trace.

Refer to Set up a multi-stack Tempo data source in Grafana for information about configuring the Tempo data source.

For information about Tempo configuration requirements, refer to the Cross-tenant query and Enable multitenancy documentation.

Next steps

• TraceQL query examples - Copy-paste query examples for common use cases
• Construct a TraceQL query - Full TraceQL syntax, scopes, and operators
• Service Graph and Service Graph view - Visualize service dependencies and RED metrics
• Span filters - Refine trace results after querying
• Configure the Tempo data source - Connection, authentication, and feature settings