Grafana Stat panel: Show the delta over a time range

A Stat panel shows one number. By default that number is the most recent value of your series, which is the right answer maybe half the time. The other half you want the change: how many orders came in since midnight, how much disk got eaten this week, how far the queue depth moved across the window you have selected in the time picker.

Grafana gives you two places to compute that, and picking the wrong one is why so many of these panels display a number nobody trusts. You can reduce the series to a single value inside the panel, or you can ask the data source for the delta directly. The second is usually correct. Here is both, with the reducer definitions quoted exactly, because the difference between Difference, Delta and Range is the whole problem.

Stat panel value options

Open a Stat panel, look at the options pane on the right and find Value options at the top. Four settings live there and only the first two matter for this.

Leave Show on Calculate. Set Calculation to something other than Last *, which is the default and the reason your panel currently shows a current value rather than a change.

Difference, Delta and Range compared

These are the three reducers people reach for, and their definitions in the Grafana calculation types reference are short enough to quote in full:

Take a gauge that starts at 100, climbs to 140, drops to 90 and ends at 120. Difference returns 20. Delta returns 70, because it counts the 40 up and the 30 up and discards the 50 down. Range returns 50, the distance from 90 to 140. Three plausible-looking numbers from one series, and only one of them answers "how much did this change".

Grafana's Delta is not the mathematical delta. If you searched for "delta" and clicked the reducer called Delta you now have a panel that reads high and never goes negative. Where Delta does something useful is on a counter that resets: because it only sums increments, a reset from 5,000 back to 0 contributes nothing negative, and the total keeps climbing. That is a real use. It is not the common one.

Delta as a word does a lot of unpaid work in monitoring. Prometheus has a delta() function, Grafana has a Delta reducer, and the two do not compute the same quantity. Neither of them is wrong about the name. Anyway.

Compute the delta in the query instead

Panel reducers work on the data Grafana received, which has already been downsampled to fit the panel width. The "first value" a reducer sees is the first point in that reduced series, not the value at the exact instant your time picker starts. On a narrow panel over a long range that gap can be hours. For anything you plan to quote to another human, compute the delta in the query.

Counters

Counters are the easy case, assuming the Prometheus data source is already wired up the way connecting Grafana to Prometheus describes. One expression covers it:

increase(http_requests_total{job="api"}[$__range])

The Prometheus function reference describes increase() as calculating the increase in the time series over the range vector, with breaks in monotonicity automatically adjusted for. That last clause is the reason to use it. When your process restarts and the counter goes back to zero, increase() knows a reset happened and accounts for it. A panel-side Difference reducer on the same data returns a large negative number and you get to explain to someone why the API served minus four thousand requests this morning.

Set Query type to Instant for this. A range query returns a series of overlapping windows and the Stat panel then reduces that series again, which works and wastes points. An instant query returns one value, which is what the panel needs. Then set Calculation to Last *, since there is only one value and you just want it shown.

Two more counter examples:

increase(node_network_receive_bytes_total{device!="lo"}[$__range])
sum(increase(nginx_http_requests_total{status=~"5.."}[$__range]))

The first gives total bytes received across the selected window, so set the unit to bytes(IEC) and the panel reads like a bandwidth report. The second gives total 5xx responses in the window, wrapped in sum() so the per-status series collapse into one number.

Gauges

A gauge goes up and down, so increase() is wrong for it. Two options depending on what you mean by change.

Net movement across the range, last value minus first value:

delta(node_filesystem_avail_bytes{mountpoint="/"}[$__range])

Prometheus documents delta() as the difference between the first and last value of each element in a range vector. Note that it extrapolates to cover the full range, so on a sparse series the result is an estimate rather than an exact subtraction. For disk space that is fine. If you need the raw subtraction with no extrapolation, do it yourself:

last_over_time(node_filesystem_avail_bytes{mountpoint="/"}[$__range])
  - first_over_time(node_filesystem_avail_bytes{mountpoint="/"}[$__range])

Peak-to-trough spread, the query equivalent of the Range reducer:

max_over_time(node_memory_MemAvailable_bytes[$__range])
  - min_over_time(node_memory_MemAvailable_bytes[$__range])

I use the max minus min form for capacity questions ("how much did free memory swing today") and the delta form for trend questions ("are we losing disk"). Both belong on a dashboard. They answer different things and putting both on the same row with clear titles beats arguing about which one is correct.

$__range and the other global variables

$__range belongs to Grafana. It is one of the global built-in variables and Prometheus never sees it. Grafana substitutes it before the query leaves the browser, so [$__range] becomes [6h] or [7d] depending on the time picker. Change the picker and every panel using it recalculates against the new window without you touching a query.

$__range is the entire window. $__interval is one step inside it, sized to the panel. Putting $__interval where you meant $__range gives you the change over the last few minutes labelled as the change over the last week, and the number comes out small enough to look believable.

Stat panel display options for a delta

Three defaults from earlier, restated here because they interact. Show defaults to Calculate and Calculation defaults to Last *. Limit defaults to 25 with a maximum of 5,000 and only appears when Show is set to All values. Show percent change is disabled by default and only applies when Show is set to Calculate.

Under Stat styles, set Graph mode to Area. That draws the sparkline behind the number so the reader can see the shape that produced it. The docs note this requires the query to return a time column, and an instant Prometheus query returns a single point and no sparkline. If you want both the accurate total and the trend line, set Query type to Both, which runs a range query and an instant query and returns combined results.

Color mode controls how far the colour spreads: None, Value, Background Gradient or Background Solid. I use Value on dense dashboards and Background Solid for a single number on a wall display, where a block of colour reads from across the room and a coloured digit does not.

Colour comes from Thresholds under standard options. For a delta, add a threshold at 0 so negative deltas take one colour and positive another. Disk free dropping is bad, orders growing is good, and the same panel type serves both if you set the base colour and the threshold colour to match the direction you care about.

There is also a Show percent change toggle, which the docs list as disabled by default and applicable only when Show is set to Calculate. Turning it on adds a small percentage under the main value. Percent change color mode then gives you Standard (green up, red down), Inverted or Same as Value, and Inverted is the one you want for a metric where growth is bad, like error count or lag. I have not found anything in the docs that says which two points the percentage is measured between, and I have not gone digging in the source to settle it either. Check it against a series whose numbers you already know before you put it in front of anyone.

Computing a delta in SQL data sources

Not everything is Prometheus. For MySQL, PostgreSQL or any of the SQL data sources, the delta is a window function and the panel just displays what you return. Set Format to Table, return one row with one numeric column, and set Calculation to Last *.

SELECT
  last_v - first_v AS delta
FROM (
  SELECT
    FIRST_VALUE(value) OVER (ORDER BY ts) AS first_v,
    LAST_VALUE(value) OVER (
      ORDER BY ts
      ROWS BETWEEN UNBOUNDED PRECEDING AND UNBOUNDED FOLLOWING
    ) AS last_v
  FROM metrics
  WHERE $__timeFilter(ts)
    AND host = 'web-01'
) t
LIMIT 1;

That explicit frame on LAST_VALUE is not decoration. The default window frame runs from the start of the partition to the current row, so LAST_VALUE without a frame clause returns the current row's value and your delta comes out as zero on every line. The panel renders perfectly. It is just showing you a zero.

$__timeFilter(ts) is the Grafana macro that expands into a BETWEEN clause matching the dashboard time picker, so this query follows the picker the same way $__range does on the Prometheus side. If your table stores a monotonically increasing counter rather than a gauge, subtract inside a SUM over positive differences instead, or the first restart will hand you a negative total.

Common causes of a wrong delta value

The first is a counter fed to a gauge reducer. If the underlying metric only ever goes up and gets reset when the process restarts, no panel-side reducer knows that. Difference returns a negative number after a restart, Delta returns something that looks right but silently drops whatever accumulated before the reset, and neither of them warns you. Use increase() and let the data source handle it.

The second is a panel time range that is not the dashboard's. Under Query options there are two settings that override the picker for that panel alone: Relative time, which the docs describe as overriding the relative time range for individual panels so they differ from the dashboard time picker, and Time shift, which shifts the panel's start and end relative to the picker. Someone sets Relative time to 24h months ago to compare against a spike, forgets and from then on that Stat panel reports a daily delta on a dashboard everyone reads as weekly. Grafana marks these panels with a small time indicator in the corner, which is easy to miss on a busy dashboard. Both overrides also only apply when the dashboard time range is relative, so an absolute range silently ignores them and the panel behaves differently again.

When the number still will not reconcile, put the same expression in Explore and read the raw series. The Grafana errors and fixes reference covers the query-side failures that present as display bugs, which is the other place to look. Explore has no reducer, no panel width and no downsampling to argue with, so whatever it returns is what your data source really holds. If Explore and the panel disagree, the problem is in the panel options.

The Stat panel gets this much attention because it is the panel people screenshot. A time series graph gets read by whoever built it. A single big number gets pasted into a channel with no context and quoted back at you in a meeting three weeks later, so somebody should be able to say which arithmetic produced it. Everything else around the panel, from variables to the layout it sits in, is in the walkthrough on Grafana dashboards, and the wider Grafana setup guide covers the pieces around all of that. The reducer list has plenty more in it than the six here, including percentiles and change counts. Reading it once saves you writing a query for something the panel already does.