.svg)
.svg)
.svg)
.svg)
.svg)
.svg)
.svg)
The rule firing and the email going out are two different systems with a routing layer between them. That is why an alert can sit in a red Firing state all afternoon while your inbox stays empty. This page walks the whole chain in the order it has to happen.
What has to exist before an email alert lands
Four things, in this order. A working [smtp] block, so Grafana can talk to a relay at all. A contact point of type Email, holding the recipient addresses. An alert rule, which is a query plus a condition on an evaluation schedule. And a notification policy that matches the firing alert and sends it to that contact point.
The first one is not optional on self-hosted Grafana. The docs say so directly: "In Grafana OSS, you must configure SMTP settings before you can enable email notifications." If you haven't done that yet, where to set up Grafana SMTP settings has the file paths and the per-provider values. Come back once a test message arrives.
Create the email contact point
In the left menu, open Alerts & IRM, then Alerting, then Notification configuration, then the Contact points tab. Click + New contact point.
Give it a name that describes who gets the mail, because you'll reuse it across rules. "Platform on-call" ages better than "High CPU alert". Pick Email as the integration and fill in Addresses, which takes a list. Email addresses are case sensitive here, and that matters more than you'd expect once mailbox aliases are involved.
A few optional settings sit lower down the form. Single email sends one message addressed to everyone instead of a separate copy per recipient, which suits a shared inbox and breaks any filter reading the To header. Subject and Message both accept notification templates, covered below. The Grafana docs on configuring the email contact point integration list every field on that form. Disable resolved message stops the "resolved" follow-up, and I leave it off, because a resolved notification is how you find out at 3am that the thing fixed itself and you can go back to sleep.
Save the contact point.
Send a test notification
Do this before you write a single alert rule. Find the contact point in the list, click Edit, then Test. Grafana offers a predefined test notification or a custom one where you supply your own annotations and labels, then Send test notification fires it.
Testing this way only works with the built-in Grafana Alertmanager. Point Grafana at an external Alertmanager and the button stops being useful, so you test through that instead. Why it doesn't proxy the test out to the external one, I don't know.
A failure reading "SMTP not configured, check your grafana.ini config file's [smtp] section" means the alerting side is fine and the mail side never started. Fixing SMTP not configured in Grafana lists every reason that flag stays false. Any other error means Grafana reached your relay and the relay said no, in which case tail the log while you retry:
sudo tail -f /var/log/grafana/grafana.logContainer users get nothing in that file by default. docker logs -f grafana is the equivalent there, and where Grafana logs live and how to read them explains why the file stays empty. Getting a real one back inside a container takes one environment variable.
Write the alert rule
Open Alerts & IRM, then Alert rules, then + New alert rule. Name it after the symptom. "Disk above 85 percent on web nodes" tells you what happened. "Disk check" tells you nothing at 3am.
Define the query and the alert condition
The query section has a Default mode for a single query with a single condition, and an Advanced mode when you need several queries and expressions chained together. Default handles most alerts. Pick your data source, then build the query in either the Builder or the Code tab.
A Prometheus example that alerts on root filesystem usage:
100 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100)That returns a time series, and an alert condition needs a single number, so Grafana adds two expressions behind it. A Reduce expression collapses the series to one value using Last, Mean, Min, Max, Sum or Count. A Threshold expression compares that number against a condition such as Is above, Is below or Is outside range, returning 1 when it matches and 0 when it doesn't. The rule fires on 1.
Set the threshold to 85 and preview it. If the preview shows Normal when you expect Alerting, the query is returning more series than you think, or the reduce function is picking a value you didn't intend. Both are easier to debug on a dashboard panel than in the rule editor, which is why the panel usually comes first and the alert second. Creating a Grafana dashboard from Prometheus data covers that half of it. Preview re-runs on every edit, so leave it open while you tune the number.
Set the evaluation behaviour
Next comes the folder the rule lives in and its labels. Labels are the routing keys, so this is where you decide how the alert gets found later. A pair like team = platform and severity = warning is enough to build every route you need.
Then the evaluation group. A group is a named bucket of rules that share an evaluation interval, and rules inside one group are evaluated sequentially. Create one with + New evaluation group, give it an interval (1m is a sensible default for infrastructure alerts). Every rule you add to that group runs on the same clock.
The pending period is the field that decides how noisy your alerting is. It's how long the condition has to stay true before the alert moves from Pending to Firing. With a 1m evaluation interval and a 5m pending period, a threshold breach has to survive five consecutive evaluations before anyone gets mailed. Set it to 0 and every transient spike pages someone.
There's also Keep firing for, which holds an alert in the firing state for a while after the condition goes false, and it's the cure for a metric that oscillates across the line. Set it to a couple of evaluation intervals on anything network-related. The field list shifts between releases, so read the Grafana docs on creating a Grafana-managed alert rule against the version you have. Below that, Configure no data and error handling decides what happens when the query returns nothing or breaks, and the default of treating no data as an alert catches dead exporters.
Choose how it notifies
The notifications section offers two modes. Default lets you pick a contact point directly on the rule, which is the fast path and fine for a handful of alerts. Advanced hands routing over to notification policies, which is what you want once you have more than a few rules and more than one recipient.
Finish with the notification message fields: Summary, Description and optionally a Runbook URL. The summary becomes the body of the email, so write it as a sentence someone can act on. Grafana's notification template reference lists everything available on the dot. {{ $labels.instance }} root filesystem at {{ humanize $values.B.Value }} percent beats "Disk high" by a mile.
Save the rule. It appears in the Alert rules list with a state of Normal. To watch it work, drop the threshold below the current value and wait out the pending period.
Route alerts with notification policies
Notification policies sit between firing alerts and contact points. Open Alerts & IRM, then Alerting, then Notification configuration and select the Notification policies tab.
One default policy exists and it catches everything that no child policy matched. It carries a Default contact point, a Group by list and a set of timers. Group wait is 30 seconds, the time it holds the first notification while similar alerts arrive. Group interval is 5 minutes, the gap before it tells you about changes to a group that already notified. Repeat interval is 4 hours, how often an unchanged firing group is sent again.
Those timer names come from Prometheus Alertmanager, which is what Grafana's unified alerting was rebuilt on top of. That is why they read oddly the first time: they were written for a component that only ever grouped and forwarded alerts and never generated any. Anyway. The defaults are reasonable, and Repeat interval is the one to change first, down to an hour for anything that wakes a human up.
Child policies are where labels earn their keep. Add one with a Matching label of team = platform pointing at your platform contact point, and every rule carrying that label routes there while everything else falls through to the default. A child policy can override the parent's grouping and timings, and Continue matching subsequent sibling nodes lets an alert match several policies instead of stopping at the first, which is how you send the same critical alert to both an inbox and a chat channel. The Grafana docs on creating notification policies set out which fields a child inherits from its parent and which it replaces.
Group by deserves one sentence of thought. Grouping on alertname and grafana_folder is the default, so twenty instances breaching the same threshold arrive as one email. Add instance to the group by list and you get twenty emails.
Silences and mute timings
A silence has a fixed start and end and matches on labels, so it's what you create when you're about to reboot a node and don't want the resulting alerts. Alerts & IRM, then Alerting, then Silences, then Create silence, set a duration, add matching labels, write a comment saying why, submit. The comment field is not decoration; a silence with no explanation is the thing you find three weeks later while wondering why nobody was alerted.
Mute timings repeat on a schedule. You define a time interval and attach it to a notification policy, and notifications matching that policy are suppressed during those windows. Non-urgent alerts muted overnight is the obvious use. Both approaches suppress the notification while the rule keeps evaluating, so the alert is still visible in the UI and still recorded in the history. Matcher syntax is the same as the policies use, and the Grafana docs on creating silences spell it out.
Give the email a subject line you can read
The default subject is functional and hard to scan in a full inbox. The Subject field on the email contact point takes a Go template with the notification data available on the dot, so you can put the useful part first.
{{ .Status | toUpper }} {{ len .Alerts.Firing }} x {{ .CommonLabels.alertname }} on {{ .CommonLabels.env }}That gives you FIRING 3 x Disk above 85 percent on production, which sorts and filters properly.
The fields available include .Status, .Alerts with its .Firing and .Resolved subsets, .GroupLabels, .CommonLabels, .CommonAnnotations and .ExternalURL.
Anything in .CommonLabels has to be common to every alert in the group, so a label that varies between them renders empty. Use .GroupLabels when you group on it.
Alerting on the latest value versus an average
Reduce defaults to Last.
That default holds up for a gauge that genuinely steps between values, like a queue depth or a systemd unit's up flag. On anything sampled it produces flapping alerts, and the mechanism is worth following because it also explains a class of NoData alerts that look like dead exporters and are not.
Prometheus scrapes on an interval, say 15 seconds. Grafana evaluates on its own interval, say 60 seconds, over a query range of the last 5 minutes. Last picks the newest sample in that range. Miss one scrape because the exporter was slow and the newest sample is now 30 seconds older, or the range briefly contains no samples at all and the reduce returns no data. On a sparse series, one that only produces a point when something happens, this is constant. The alert flaps between Firing, Normal and NoData, and each transition is an email.
Do the smoothing in the query. For a counter, wrap it in rate() over a window of at least four scrape intervals, which for a 15 second scrape means rate(http_requests_total[1m]) as an absolute floor and [5m] in practice. For a noisy gauge, avg_over_time(node_load1[5m]) does the same job. Then leave Reduce on Last, because the averaging already happened where it belongs and the reduce is only collapsing a smoothed series to a number.
None of this shows up while you are building the rule. A rule you preview on a quiet afternoon evaluates cleanly, and the flapping starts the first time an exporter has a bad ten minutes at an hour nobody is watching. To see it before your users do, copy the rule, set the pending period to 0 on the copy and point it at a mailbox nobody reads for a day.
The second half of the fix is the pending period. An average over 5 minutes with a pending period of 5 minutes means the condition has to hold across roughly ten minutes of real data before anyone is notified. That sounds slow until you compare it against the alternative, which is an inbox everyone has learned to ignore. For latency and error-rate alerts I use a shorter query window with a longer pending period, since I'd rather detect a sharp change quickly and confirm it than average a real incident away. Whatever you pick, write it down in the rule's description, because the next person will ask why the window is 5 minutes and not 1.
Grafana Cloud does not need SMTP
None of the mail setup applies on Grafana Cloud. The docs state it plainly: "In Grafana Cloud, SMTP configuration is not required." Email contact points work out of the box there because Grafana runs the mail infrastructure, and the [smtp] section is irrelevant. That difference sits alongside the rest in the comparison of Grafana Cloud against self-hosted Grafana. Everything else on this page, the rules and the routing, is identical between the two.
On a self-hosted instance the sending path is yours, which means the alert about your mail server going down arrives by email. Build a second contact point on a different transport, a webhook into chat or a push service, and route your highest severity to both. The roundup of common Grafana errors and how to fix them collects the other symptoms that turn up during setup, and the wider picture sits in the complete guide to running Grafana. That is the whole chain, from the SMTP block through to the routing tree.
