.svg)
.svg)
.svg)
.svg)
.svg)
.svg)
.svg)
Most Grafana errors are one of about a dozen things wearing different clothes. The error strings are precise once you know where in the code they come from. This page maps each string to its real cause and the first command to run.
| Symptom | Real cause | First command |
|---|---|---|
| Test email fails, log says SMTP not configured | [smtp] enabled is false at runtime | grep -n '^enabled' /etc/grafana/grafana.ini |
| Service will not start after editing SMTP | from_address is empty or malformed | journalctl -u grafana-server -n 30 |
| admin / admin rejected on a fresh install | Password already changed, or a provisioned override | grafana cli admin reset-admin-password <new> |
| Login blocked after repeated failures | Brute force protection, 5 attempts | grep 'consecutive incorrect' /var/log/grafana/grafana.log |
| Grafana will not bind, port 3000 taken | Another process owns the port | sudo ss -tlnp | grep :3000 |
| Starts, then exits within a second | Ownership of the data or log directory | ls -ld /var/lib/grafana /var/log/grafana |
| Blank page with a "failed to load" message | root_url or serve_from_sub_path | curl -sI http://127.0.0.1:3000/login |
| Datasource test returns a connection error | Wrong address from Grafana's own network namespace | curl -s http://localhost:9090/-/ready |
| Panel shows No data, query works elsewhere | Time range, timezone or step interval | Open panel, Query inspector, Query tab |
| grafana.log is empty in Docker | Entrypoint forces console-only log mode | docker logs --tail 50 grafana |
| Random 500s under load, SQLite backend | Write contention on grafana.db | grep -c 'database is locked' /var/log/grafana/grafana.log |
| Installed plugin does not appear | Unsigned plugin, blocked at load | grep -i signature /var/log/grafana/grafana.log |
Every entry below assumes you can read the log. Grafana log file locations and log levels is the prerequisite if you cannot, because guessing at Grafana errors without the log is a slow way to spend an evening. The path differs by install method.
SMTP not configured, check your grafana.ini config file's [smtp] section
In pkg/services/notifications/mailer.go the message comes from a single guard: if Cfg.Smtp.Enabled is false, return ErrSmtpNotEnabled. That is the only place it appears. A wrong hostname, a rejected password, a blocked outbound port and a TLS mismatch all produce different errors from the dialer, later in the send path.
So the error means one thing: at the moment Grafana loaded its config, enabled in [smtp] was false. Nine times out of ten the line is present in /etc/grafana/grafana.ini but still carries the leading semicolon that ships with the sample config, and INI treats anything after that semicolon as a comment.
sudo grep -n -A3 '^\[smtp\]' /etc/grafana/grafana.iniRemove the semicolon from ;enabled = false, set it to true, restart. The fix for SMTP not configured in Grafana covers the other six reasons that flag stays false, including the environment variable form. Most of the time it is the semicolon.
invalid email address for SMTP from_address config
This one kills the process at startup. Grafana refuses to construct its notification service and exits, which is why the message lands in journalctl and never reaches grafana.log.
sudo journalctl -u grafana-server -n 40 --no-pagerThe validation runs on Cfg.Smtp.FromAddress and requires something that parses as an email address. An empty value fails. So does grafana@localhost without a dot in the domain, and so does anything with a stray quote or trailing space picked up from a copy and paste. The shipped default is [email protected], which passes validation, so if you are hitting this you have edited the line. Every key in that section is listed in the Grafana SMTP settings reference.
[smtp]
enabled = true
host = smtp.example.com:587
from_address = [email protected]
from_name = GrafanaSet a sender your relay will accept. Passing validation and being deliverable are separate questions, and a mismatched envelope sender bounces somewhere you are not watching.
admin / admin is rejected on a new instance
The documented default is username admin, password admin, at http://localhost:3000, with a prompt to change the password on first login. When that combination fails on something you just installed, there are two realistic explanations. Someone completed the first-login prompt already, or the instance was provisioned with GF_SECURITY_ADMIN_PASSWORD set, which silently replaces the default before you ever see the login form.
Reset it from the CLI on the host:
sudo grafana cli --homepath "/usr/share/grafana" admin reset-admin-password <new password>
sudo systemctl restart grafana-serverIn Docker, run the same command inside the container with docker exec. The Grafana default login and password reset guide has the full detail, including the provisioning case. A reset that reports success while login still fails is the lockout in the next section.
too many consecutive incorrect login attempts for user - login for user temporarily blocked
That is the log line. In the browser you get something far less helpful, either "Invalid username or password" or "Login temporarily blocked", so the log is where the truth is. Grafana's brute force protection counts failures per username and separately per IP address, and the threshold is five:
[security]
disable_brute_force_login_protection = false
brute_force_login_protection_max_attempts = 5
disable_username_login_protection = false
disable_ip_address_login_protection = trueIP address protection is off by default and username protection is on. The block is time based and clears on its own, so waiting is the fix. Disabling the protection to get in faster is a bad trade on anything reachable from the internet, for reasons covered in Grafana security best practices and hardening.
Confirm you are looking at a lockout rather than a wrong password:
sudo grep -i 'consecutive incorrect login attempts' /var/log/grafana/grafana.log | tail -5Grafana will not start and port 3000 is already in use
Grafana defaults to HTTP port 3000 via [server] http_port. When something else has it, the startup log carries a bind failure mentioning "address already in use" and the service dies immediately. Find the squatter:
sudo ss -tlnp | grep ':3000'The usual suspects are a Node dev server, a previous Grafana that systemd thinks is stopped, or another monitoring tool that also picked 3000. Stop the other process, or move Grafana. Moving is one line plus a restart.
[server]
http_port = 3001Update root_url and your reverse proxy in the same change, or the startup error becomes the blank page described below. Changing the Grafana default port has the port options and the proxy config. Restart after either edit.
Grafana starts and then exits within a second
Almost always ownership. The service runs as the grafana user and needs write access to its data directory and its log directory.
ls -ld /var/lib/grafana /var/log/grafana /var/lib/grafana/plugins
sudo journalctl -u grafana-server -n 30 --no-pagerThe packaged layout expects grafana:grafana on all of them:
sudo chown -R grafana:grafana /var/lib/grafana /var/log/grafana
sudo systemctl restart grafana-serverTwo extra traps. The systemd unit ships with ProtectHome=true and ProtectSystem=full, so a data or log path relocated under /home is unreachable regardless of permissions. Docker fails the same way from a different direction: the container process runs as uid 472 and gid 0, and a bind mount created by your login account is not writable by that pair. A named volume sidesteps it.
If you're seeing this Grafana has failed to load its application files
The backend is fine. This message is served by Grafana's own HTML shell when the browser could not fetch the JavaScript bundle, and the page helpfully lists the causes, the first two of which account for nearly everything: "This could be caused by your reverse proxy settings" and "If you host grafana under a subpath make sure your grafana.ini root_url setting includes subpath. If not using a reverse proxy make sure to set serve_from_sub_path to true."
Confirm the backend is healthy before touching config:
curl -sI http://127.0.0.1:3000/login
curl -s http://127.0.0.1:3000/api/healthA 200 from both means the problem is entirely in the path between browser and Grafana. Serving at https://example.com/grafana/ needs both settings agreeing:
[server]
domain = example.com
root_url = https://example.com/grafana/
serve_from_sub_path = trueThe trailing slash on root_url matters.
On the proxy side, the location block has to preserve the prefix that root_url promises. An nginx proxy_pass with a trailing slash rewrites /grafana/public/build/x.js to /public/build/x.js, Grafana has nothing at that path, and it answers with the HTML shell again. Drop the trailing slash on proxy_pass and it works.
That trailing slash rule is one of the older footguns in nginx and it still catches people who have been writing nginx configs for a decade. With a slash on proxy_pass, the matched location prefix is replaced by the URI you gave. Without one, the request URI is passed through whole. The nginx documentation describes both behaviours correctly and it is still a single character in a file most people copied from a blog post. Once it has bitten you, you check for it first.
The Prometheus datasource test returns a connection error
The Save and test button reports what Grafana's backend saw, not what your browser can reach. Grafana resolves the URL from its own network position, so on a bare host that is localhost and in a container it is the container.
curl -s http://localhost:9090/-/ready
sudo ss -tlnp | grep ':9090'If that works from the shell but the test fails, check where Grafana is running. In Docker Compose, http://localhost:9090 from inside the Grafana container means the Grafana container, so use the service name: http://prometheus:9090. On a host with Prometheus bound to 127.0.0.1 and Grafana in a container, the container cannot reach it at all until you bind Prometheus to the bridge address or run both on the same network.
The other frequent cause is a firewall rule between two machines, and it shows up as a timeout. Refused means nothing is listening, timeout means something is dropping packets. Building the connection properly, with the query and dashboard side included, is covered in connecting Prometheus to Grafana and building a dashboard.
A panel shows No data but the query works in Prometheus
Four things produce this, in rough order of frequency. Start with the panel time range, since Prometheus's own UI defaults to the last hour while your dashboard might be showing a window where the series genuinely has no samples. Then check the dashboard timezone against the browser and the server, because a panel set to a fixed timezone on a UTC host can be querying a range that has not happened yet.
Open the panel menu, choose Inspect, then Query, and read the actual request Grafana sent. The start and end timestamps in that request are the ground truth and they settle the timezone question in about ten seconds.
The third cause is step interval. Grafana calculates the query step from panel width and time range, and a range function with a window shorter than the scrape interval returns nothing. $__rate_interval fixes it, since that macro is built to stay above the scrape interval, and the Prometheus query editor docs explain what it resolves to:
rate(http_requests_total[$__rate_interval])The fourth is the Min step setting on the Prometheus datasource. The Grafana docs suggest setting it to match your scrape interval, so 15s against a 15 second scrape, which stops Grafana asking for a resolution the data does not have. Leaving it empty makes the query use $__interval.
grafana.log is empty inside a Docker container
Not broken. The official image's entrypoint ends its exec line with cfg:default.log.mode="console", which drops file mode from the default configuration.
docker logs -f grafanaThat is where your logs are. The /var/log/grafana directory exists anyway, because GF_PATHS_LOGS is baked into the image. If you want a file in it, the entrypoint value sits in Grafana's defaults layer, so an environment variable overrides it:
docker run -d --name=grafana -p 3000:3000 \
-e "GF_LOG_MODE=console file" \
-v grafana-storage:/var/lib/grafana \
-v grafana-logs:/var/log/grafana \
grafana/grafanaKeep console in the list so docker logs keeps working. The precedence rules behind the override are explained in running Grafana with Docker Compose.
database is locked
SQLite is Grafana's default backend and it is a good one for a single instance with a handful of users. It is a single writer, so concurrent writes queue, and past a certain amount of activity they start timing out with the SQLite driver's database is locked error. Alerting is usually what tips it over: rule evaluations write state on every tick, and a hundred rules on a 30 second interval is a lot more write traffic than the dashboards themselves ever generate.
sudo grep -c 'database is locked' /var/log/grafana/grafana.logA non-zero count that grows over a day means you have outgrown the default. Two responses. Write-ahead logging buys headroom and costs nothing:
[database]
type = sqlite3
wal = truePast that, move to PostgreSQL. Grafana's own defaults hint at the boundary: high_availability defaults to true and the comment says to set it false only when running a single instance. My rule is that anything with real alerting, more than a few concurrent users or a second Grafana node in its future goes on Postgres from day one, because migrating a live instance later means an export and a reimport.
An installed plugin does not appear in the UI
Grafana verifies plugin signatures at load time and refuses unsigned plugins by default. The plugin lands on disk, the load step rejects it, and the UI shows no sign that anything happened. The log does:
sudo grep -i -E 'signature|unsigned|plugin' /var/log/grafana/grafana.log | tail -20For a plugin you built yourself or one that is genuinely unsigned and genuinely trusted, allow it by ID:
[plugins]
allow_loading_unsigned_plugins = my-custom-datasource,another-plugin-idThe setting takes a comma separated list of plugin IDs. There is no value meaning allow everything, by design. Restart after changing it.
A plugin from the official catalogue that shows as unsigned usually had its download interrupted. Remove the directory under /var/lib/grafana/plugins and reinstall with grafana cli plugins install <plugin-id>. To see which check failed, turn on the loader's own logging:
[log]
filters = plugin.loader:debugInformation to collect before you ask for help
Forum posts that get answered in an hour look different from ones that sit for a week, and the difference is entirely what got pasted. Start with the version, from the shell or the UI footer:
grafana cli -v
dpkg -l grafana | tail -1Then raise the level for the one subsystem involved. Global debug on a live instance produces thousands of lines a minute and buries the twenty that matter:
[log]
level = info
filters = notifications:debugUseful logger names include sqlstore for database problems, plugin.loader for plugins, provisioning for dashboards and datasources that arrive from files, ngalert for alerting and authn.password for login trouble. Restart, reproduce the problem once, then pull the window around it:
sudo journalctl -u grafana-server --since "5 minutes ago" --no-pager | tail -60Paste the lines with level=error plus the ten before them, since the error is usually the consequence and the cause is above it. Grafana's troubleshooting documentation and the configuration reference are the two pages to have open while you write the post. Include your install method, the config keys you changed with values redacted and the exact output of the command that failed. Set the filter back to empty afterwards.
