The Dashboard That Spins While You Panic
It’s 2 AM. Something is on fire, you open the Grafana dashboard that is supposed to tell you what, and you get a spinner. Twelve seconds later, half the panels render and the other half say “Query timeout”. The metrics are fine. Prometheus has the data. Your dashboard is just asking it to redo a pile of math every time anyone looks.
Recording rules fix that. A recording rule runs a PromQL expression on a schedule and stores the result as a new time series. The panel then reads one cheap, already-aggregated series instead of crunching thousands of raw ones on every refresh. You pay the cost once per evaluation interval instead of once per viewer per panel per refresh.
This is the combo to learn if you already have Prometheus and Grafana running (if you don’t, start with Prometheus and Grafana setup). The rules are twenty lines of YAML. The skill is knowing which queries deserve one.
Full example: Clone the working files at github.com/KingPin/sumguy-examples/observability/prometheus-recording-rules
What a Recording Rule Actually Does
Picture moving a couch with a forklift. It works, but you don’t hire the forklift every time someone wants to sit down. You move the couch once and everybody sits. A recording rule is the single move. The dashboard is everyone sitting.
Without a rule, a panel that shows CPU utilisation sends something like this on every refresh:
1 - sum without (cpu, mode) (rate(node_cpu_seconds_total{mode="idle"}[5m])) / sum without (cpu, mode) (rate(node_cpu_seconds_total[5m]))That query touches every CPU core and every mode on every machine, then runs rate() over a window, then aggregates. Do it for 6 panels, 4 viewers, a 30-second refresh, and a 24-hour range, and Prometheus does a lot of repeated work. The official docs say the same thing: precomputing is “especially useful for dashboards, which need to query the same expression repeatedly every time they refresh.”
With a rule, the expression runs once per interval and the panel queries the output name instead:
instance:node_cpu_utilisation:ratio_rate5mOne series per machine. No rate(), no aggregation, no per-core fan-out at query time.
The Kit: Four Rules You Can Run
The companion repo has a Compose file with Prometheus v3.15.1, node_exporter v1.12.1, and Grafana 13.3.0, plus the rules below. Here is the rule file.
groups: - name: node_cpu interval: 15s limit: 500 rules: # Step 1: per-mode CPU seconds per second, cores summed away. - record: instance_mode:node_cpu_seconds:rate5m expr: sum without (cpu) (rate(node_cpu_seconds_total[5m]))
# Step 2: reads step 1. Same group, so it runs after step 1, same timestamp. # Numerator and denominator are aggregated separately, then divided. # ignoring (mode) because the numerator keeps mode="idle" and the denominator has no mode label. - record: instance:node_cpu_utilisation:ratio_rate5m expr: |2 1 - instance_mode:node_cpu_seconds:rate5m{mode="idle"} / ignoring (mode) sum without (mode) (instance_mode:node_cpu_seconds:rate5m)
- name: node_memory_network interval: 15s rules: - record: instance:node_memory_utilisation:ratio expr: 1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes
- record: instance:node_network_receive_bytes:rate5m expr: sum without (device) (rate(node_network_receive_bytes_total{device!~"lo|veth.*|docker.*|br-.*"}[5m]))And the Prometheus config that loads it:
global: scrape_interval: 15s evaluation_interval: 1m # default; groups can override with their own interval
rule_files: - /etc/prometheus/rules/*.yml
scrape_configs: - job_name: node static_configs: - targets: ["node_exporter:9100"] - job_name: prometheus static_configs: - targets: ["localhost:9090"]Start it:
docker compose up -dGive it about a minute, because rate(...[5m]) needs a few scrapes. Then ask Prometheus for the recorded series:
curl -s -G localhost:9090/api/v1/query \ --data-urlencode 'query=instance:node_cpu_utilisation:ratio_rate5m'On my run that returned one series for the node_exporter instance with a value around 0.065 (6.5% CPU on an idle laptop). Memory and network came back the same way. Open http://localhost:9090/rules and you’ll see each group, its last evaluation time, and per-rule health.
Naming: level:metric:operations
Those colons in the names are not decoration. Prometheus reserves colons in metric names for recording rules, and the official naming page says to use the form level:metric:operations:
- level is the aggregation level, meaning the labels that remain on the output.
instance_modemeans the series is still split by instance and mode. - metric is the source metric, unchanged except that you strip
_totalfrom counters when you applyrate(). - operations lists what you applied, newest first.
rate5mmeans a 5-minute rate.ratio_rate5mmeans a ratio of rates.
Follow it and your 2 AM self will thank you. When you see job:http_requests:rate5m in a panel, you know the labels, the source, and the math without opening the rule file.
The docs also give two aggregation rules worth tattooing somewhere. First, always use without and list the labels you are dropping, so job and friends survive. Second, never average a ratio or an average. Aggregate the numerator and denominator separately, then divide. That’s why the CPU rule above has two steps instead of one avg().
Chaining Rules and Evaluation Order
The CPU example has two rules in one group, and the second reads the first one’s output. That works because the docs say rules inside a group “are run sequentially at a regular interval, with the same evaluation time.” Rule two sees rule one’s fresh result.
Groups, on the other hand, run independently of each other. If you put step 2 in a different group, nothing guarantees step 1 finished first, and you get a value that is one interval stale or missing. Keep dependent rules in the same group, in order.
Chaining is also how you build summary layers. Record per-instance values first, then record a fleet-level series from those. In the kit, that’s one more rule in the node_memory_network group, directly after the instance-level network rule:
- record: job:node_network_receive_bytes:rate5m expr: sum without (instance) (instance:node_network_receive_bytes:rate5m)It reads the instance-level output from the same evaluation and reuses that rule’s device filter. Never define the same record: name in two groups: both write the same series, and the line jumps between their values. A fleet-wide “total traffic” panel then reads one series per job, not one per interface per machine.
Picking an Interval
Each group has its own interval. If you leave it out, the group uses global.evaluation_interval, which defaults to 1 minute. The kit sets interval: 15s on its groups to match the scrape interval, so a panel gets a new point every scrape.
Two rules of thumb:
- Don’t evaluate faster than you scrape. A 5-second rule over a 15-second scrape just recomputes the same input.
- Match the evaluation interval to the panel step you actually look at. A dashboard that shows 24 hours at a 1-minute step gains nothing from a 5-second rule.
The range inside rate() is a separate decision. It needs enough samples to compute a rate, and the common guidance is at least four times the scrape interval. With a 15-second scrape, [1m] is the floor and [5m] is a safe default. Short ranges are twitchy, and one missed scrape makes them lie.
Using the Recorded Series in Grafana
In a panel, switch to the code editor and type the recorded name. That’s it. No special data source, no plugin:
instance:node_cpu_utilisation:ratio_rate5m{instance=~"$instance"}Your dashboard variables (see Grafana dashboard variables) still work, because the instance label survives on the output.
One gotcha trips people. Grafana’s $__rate_interval variable adapts the rate() window to the panel’s step and the data source’s scrape interval. It is a Grafana feature and it only exists inside Grafana queries. A recording rule runs inside Prometheus, so it needs a fixed range such as [5m]. If you paste a panel query into a rule file, you must replace $__rate_interval with a literal duration, and the name should say which one you picked (rate5m).
The flip side: a panel reading ...:rate5m always shows a 5-minute rate, whatever the zoom level. For a trend panel, that’s fine. For a “what happened in the last 30 seconds” panel, query the raw counter with $__rate_interval instead.
Check Your Work with promtool
Never ship a rule you haven’t parsed. promtool lives inside the Prometheus image, so you don’t need to install anything:
docker run --rm -v "$PWD":/w -w /w --entrypoint promtool \ prom/prometheus:v3.15.1 check rules rules/*.ymlChecking rules/node.yml SUCCESS: 4 rules foundSyntax is only half of it. You also want to know the math is right, and promtool test rules lets you feed in fake series and assert what the rules produce:
rule_files: - ../rules/node.yml
evaluation_interval: 15s
tests: - interval: 15s input_series: # Two cores. Each core is idle 0.5 s per second, so utilisation is 50%. - series: 'node_cpu_seconds_total{instance="a:9100", cpu="0", mode="idle"}' values: '0+7.5x40' - series: 'node_cpu_seconds_total{instance="a:9100", cpu="1", mode="idle"}' values: '0+7.5x40' # Busy time: 7.5 s per 15 s scrape per core. - series: 'node_cpu_seconds_total{instance="a:9100", cpu="0", mode="user"}' values: '0+7.5x40' - series: 'node_cpu_seconds_total{instance="a:9100", cpu="1", mode="user"}' values: '0+7.5x40' - series: 'node_memory_MemTotal_bytes{instance="a:9100"}' values: '1000x40' - series: 'node_memory_MemAvailable_bytes{instance="a:9100"}' values: '250x40' promql_expr_test: - expr: instance:node_cpu_utilisation:ratio_rate5m eval_time: 8m exp_samples: - labels: 'instance:node_cpu_utilisation:ratio_rate5m{instance="a:9100"}' value: 0.5 - expr: instance:node_memory_utilisation:ratio eval_time: 8m exp_samples: - labels: 'instance:node_memory_utilisation:ratio{instance="a:9100"}' value: 0.75docker run --rm -v "$PWD":/w -w /w --entrypoint promtool \ prom/prometheus:v3.15.1 test rules tests/*.yml SUCCESSThis test earned its keep while I wrote the kit. My first version of the CPU rule divided a vector that still had a mode="idle" label by one that had no mode label. PromQL silently matched nothing, and the test reported got: nil. Without the test, the panel would have shown “No data” and I’d have blamed Grafana. The fix was ignoring (mode) on the division.
Watching the Rules Themselves
Rules are workloads. Prometheus exposes metrics about them, and the kit scrapes Prometheus itself so you can see them. These names exist in v3.15.1 (I queried them in the lab):
prometheus_rule_group_last_duration_secondsis how long the last evaluation of a group took. In the lab, both groups took under a millisecond per evaluation. Those are tiny numbers, on a tiny lab.prometheus_rule_group_interval_secondsis the configured interval, 15 in the kit.prometheus_rule_evaluation_failures_totalcounts failed evaluations. It was 0 for both groups.
The alarm to set: duration approaching the interval. If a group takes 14 seconds on a 15-second interval, it’s about to skip evaluations, and your panels grow gaps. Either split the group, raise the interval, or simplify the expression.
I’m not going to quote a speedup number. A one-machine lab with a few hundred series answers in about a millisecond either way, and anything I printed would be noise. The effect shows up when the raw query fans out over thousands of series and a long time range. Measure that on your own data with the query inspector in Grafana (or the query stats in Prometheus) before and after.
Gotchas That Bite After You Ship
No backfill. A recording rule only writes data from the moment Prometheus starts evaluating it. Your new panel will be empty for the past. There’s an escape hatch: promtool tsdb create-blocks-from rules generates TSDB blocks for new rules from existing data, and it takes a --start flag plus the rule files. It’s meant for one-off backfills, and the blocks have to be moved into the Prometheus data directory, so read its help before you point it at anything you care about. It also can’t backfill a rule that reads another rule being backfilled, such as step 2 of the CPU chain. Backfill step 1, move those blocks in, then run it again for step 2.
Staleness. When the source series disappears (a target goes down, a container is removed), the recorded series stops too, and Prometheus marks it stale like any other series. That’s what you want, but it means a recording rule is not a way to keep a graph line alive past a dead target.
Labels. Rules can add or overwrite labels with a labels: block, at group level or per rule. Rule-level labels win on a collision. Use it for tags like team: infra, not for fixing a messy aggregation.
The limit field. limit: 500 on a group caps how many series a recording rule may produce (0 means no limit). When the limit is exceeded, all the series from that rule are discarded and the evaluation counts as an error. It’s a seatbelt against a bad label turning one rule into a cardinality explosion.
Reload without restart. Send SIGHUP to the Prometheus process, or hit the reload endpoint if you started it with --web.enable-lifecycle. Prometheus only applies the change if every rule file is well formed, so a typo leaves the old rules running.
Late data. Newer Prometheus versions have query_offset on a group (and rule_query_offset globally) that shifts the evaluation timestamp into the past. The docs describe it for setups where data lands late, for example when Prometheus is a remote-write target. A plain scrape setup doesn’t need it.
When NOT to Write a Recording Rule
Rules cost storage, memory, and evaluation time. Skip them when:
- The query is already cheap. A raw gauge selector like
node_memory_MemAvailable_bytesis one series per machine. Recording a copy of it just doubles the series count. (The memory rule in the kit exists to show the pattern and the ratio. On a real fleet, you’d probably skip it.) - You run it once. Ad-hoc investigation queries should stay ad-hoc. Nobody wants a rule file full of one-off debugging.
- The output has the same cardinality as the input. If a rule keeps every label and just wraps a function around it, you’ve copied the data instead of shrinking it. The win comes from aggregating labels away (
sum without (cpu)) or collapsing many series into few. - The panel needs an adaptive range. As covered above, a
$__rate_intervalpanel can’t use a fixed-range rule. - You haven’t measured. Look at the slowest panels first. Grafana’s query inspector shows the time per query. Rule the top three offenders, not everything.
A good heuristic: if a panel takes seconds to render and the query has rate() or histogram_quantile() inside a sum, it’s a candidate. If the panel is instant, leave it alone.
Cleanup
docker compose down -vThe SumGuy Take
Start with the three slowest panels on your busiest dashboard. Write one rule each, name it level:metric:operations, add a promtool test rules case, and ship. Skip the grand plan to precompute your whole metric catalog. Over-engineering a rule layer is its own flavor of 2 AM pain.
Common Questions
Do Prometheus recording rules backfill old data?
No. A recording rule only writes samples from the moment Prometheus starts evaluating it, so panels start empty for earlier periods. To fill history, run promtool tsdb create-blocks-from rules against existing data, then move the generated blocks into the Prometheus data directory. Check its help output first.
How often should a recording rule run?
Match the evaluation interval to your scrape interval, and no faster. A group without interval uses global.evaluation_interval, which defaults to 1 minute. With a 15-second scrape, a 15-second group interval gives dashboards a fresh point per scrape. Faster than the scrape just recomputes identical input.
Can I use $__rate_interval in a Prometheus recording rule?
No. The $__rate_interval variable is replaced by Grafana before a query is sent, and Prometheus rule files never go through Grafana. A recording rule needs a literal range such as [5m]. Put that duration in the rule name (rate5m) so panel authors know what they are reading.
Do recording rules in one group run in parallel?
No, not by default. Rules in a single group run sequentially in file order, with the same evaluation timestamp, so a later rule can read an earlier rule’s output. The opt-in concurrent-rule-eval feature flag parallelizes only rules with no dependencies. Separate groups run independently, so keep dependent rules in the same group.
How do I know if a recording rule group is too slow?
Compare prometheus_rule_group_last_duration_seconds with prometheus_rule_group_interval_seconds. If the duration approaches the interval, the group will start missing evaluations and panels will show gaps. Split the group, raise its interval, or simplify the expression. The /rules page in the Prometheus UI shows per-rule evaluation time.
