.svg)
.svg)
.svg)
.svg)
.svg)
.svg)
.svg)
Grafana is a query and visualisation layer that connects to systems already holding your data. You open a dashboard, Grafana sends a query to each data source behind it, then draws whatever comes back. The most repeated misconception about it is that Grafana stores your metrics. It does not. There is no time series database in the product and no endpoint accepting samples, so when the system it queries is empty or unreachable, your dashboard is blank no matter how well built it is.
What Grafana's own database does hold is dashboards, users, folders, permissions, alert rules and data source definitions. That one distinction shapes almost everything else about running it: what you back up, how it fails, how you size the server and which half of a broken setup you go and look at first. This guide covers what Grafana does, the routes to getting it running, how to connect data to it and what operating it looks like after the first week.
What Grafana does and what it doesn't store
Four things make up the product. Data sources are connection definitions, each one a type plus a URL plus credentials. Panels are visualisations, each carrying one or more queries written in the query language of whatever it points at. Dashboards are collections of panels sharing a time range and a set of template variables. Alert rules are queries with a condition and an evaluation schedule attached, plus the routing that decides who hears about it.
Everything you see on a Grafana screen is a query executed at view time. Open a dashboard with thirty panels over a ninety day range and Grafana fires thirty queries at your data sources right then, waits for the results and renders them. Nothing was precomputed, and nothing is cached by default beyond the browser session.
Four consequences follow from that. Dashboard performance is query performance, so a slow dashboard is nearly always a slow data source. Deleting a dashboard deletes no data. Losing your Grafana instance loses every dashboard and alert rule while losing not one sample of the metrics themselves. And a Grafana with a broken data source looks identical to a Grafana with an empty one, which is why the Save & test button on a data source page matters more than it appears to.
The database behind Grafana is SQLite by default, sitting at /var/lib/grafana/grafana.db on a package install. The supported alternatives are MySQL from 8.0 and PostgreSQL from 12. SQLite holds up while one instance serves a small team. It stops holding up once alerting gets busy, since every rule evaluation writes state and SQLite admits one writer at a time.
Where Grafana sits in a monitoring stack
A working setup has four jobs in it and Grafana does one of them.
| Job | Typical component | What it owns |
|---|---|---|
| Collection | Node Exporter, an application's /metrics endpoint, Alloy | Turning a machine or a process into numbers somebody can read |
| Storage and query | Prometheus, Loki, PostgreSQL, Elasticsearch | Keeping the samples and answering queries about them |
| Presentation and access control | Grafana | Dashboards, users, permissions, alert rules |
| Delivery | An SMTP relay, a chat webhook, a paging service | Getting a notification to a human |
Prometheus and Grafana hold two adjacent rows of that table, which is why every Grafana tutorial turns into a Prometheus tutorial four paragraphs in. You cannot demonstrate the third row without the second. Grafana against Prometheus works through the architectural split, including the cases where you genuinely only need one of them. Read that first if you are still choosing components.
The fourth row goes missing until an alert does not arrive. Self-hosted Grafana has no mail relay inside it, so notifications depend on infrastructure you supply, and configuring that is a separate exercise from writing the alert rule.
How to install Grafana
The current release line is 13.x. One naming change across the install surface catches people: the binary is now invoked as grafana server and grafana cli, two words each, while the systemd unit kept its old name of grafana-server.service. The hyphenated grafana-cli still runs and prints a deprecation warning telling you to stop using it. Same string, two different things, and mixing them up costs an afternoon.
Package install from the APT repository
Use this on any VPS where Grafana is the main job. I pick it over the alternatives without much deliberation, because every path it creates matches what the official documentation assumes, which makes every other page you read afterwards easier to follow. You get a systemd unit, a service user that is not root, config at /etc/grafana/grafana.ini, data at /var/lib/grafana, logs at /var/log/grafana and an upgrade path through the package manager you already run.
Repository setup is four commands:
sudo apt-get install -y apt-transport-https wget gnupg
sudo mkdir -p /etc/apt/keyrings
sudo wget -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
sudo chmod 644 /etc/apt/keyrings/grafana.ascAnything telling you to pipe that key into apt-key add predates modern Ubuntu, where the command has been removed. Then add the repository and install:
echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install grafanaThe official Debian and Ubuntu installation docs cover both the grafana package, which is the open source build, and grafana-enterprise, the Enterprise one from the same repository. Start with the first unless you already hold a licence.
The systemd start, the firewall rules and the reverse proxy are in installing Grafana on an Ubuntu VPS, which is the full walkthrough with the commands in order. If you would rather skip the repository dance entirely, LumaDock's one-click Grafana VPS template deploys that same package layout with the service already running. The paths are identical either way.
Docker and Docker Compose
Docker earns its place when Grafana shares a box with other services. Upgrades become an image tag change, and the whole arrangement is one file you can read in thirty seconds. The cost is a permissions model you have to learn once, and a log file that will not be where you expect it.
Inside the image Grafana runs as uid 472, gid 0. Bind-mount a directory you made from your own shell and it cannot write to it, which surfaces at startup as a complaint about the database file. Running Grafana with Docker Compose has a working compose file, the provisioning directory layout and the volume arrangement that avoids exactly that. The entrypoint also appends cfg:default.log.mode="console", so /var/log/grafana/grafana.log gets created in the container and then sits at zero bytes forever while docker logs carries everything. Both behaviours are deliberate.
On a single-purpose box, containers add indirection over networking and file ownership and hand back nothing. The package install is simpler there.
Grafana Cloud
The managed option. Its free tier is not a trial, it covers a small estate comfortably, and the two limits that decide your bill are active metric series and ingested log volume. Where the crossover sits, and what self-hosting really costs in hours, is worked out in Grafana Cloud against self-hosted Grafana. Grafana Cloud also sends alert email for you, so the entire SMTP section below is irrelevant there.
First login, admin password and port 3000
Grafana binds TCP port 3000 on all interfaces by default, so the moment the service starts, anyone scanning your IP finds a login page. Open http://YOUR_SERVER_IP:3000 and sign in with username admin and password admin. Grafana prompts you to change it immediately. Take that prompt, because a published default password behind a public port gets found by scanners within hours.
Setting admin_password in grafana.ini after the first start changes nothing. The value is a seed, read only when Grafana needs to create the initial admin row in its database, and after that row exists the config key is ignored. On a running instance you change the password in the UI or reset it from a shell:
sudo grafana cli --homepath "/usr/share/grafana" admin reset-admin-password <new password>The Grafana default login and how to change it collects the Docker, Helm and external-database variants of that command, plus what to do when admin / admin is rejected on something you just installed. Brute force protection blocking you after five failed attempts is a common cause and rarely the first guess.
Port 3000 itself lives in [server] http_port, and the neighbouring keys matter more than the port number does. http_addr controls the interface, and setting it to 127.0.0.1 makes the instance unreachable from outside the box regardless of firewall state, which is what you want the moment nginx or Caddy is in front. root_url is the full public address Grafana believes it lives at. Redirects, share links and OAuth callbacks all get built from that value, and the incoming Host header is ignored, so a wrong one produces symptoms with no obvious connection to a URL. The Grafana default port 3000 and how to change it covers the whole family of settings, including the subpath case and the Docker port mapping trap. Change the number itself only if something else on the box already holds 3000.
Since we are on 3000: it is also the default for Create React App, Next.js in dev, Rails with some setups and half the Node tooling written since 2015. On a developer's laptop that collision is constant, and on a server it is the reason a second service sometimes refuses to start after you install Grafana. Nothing to fix, just something to know before you spend twenty minutes on it.
Two independent barriers is the arrangement to run: bind to loopback in the config, and keep a firewall rule that never allowed 3000 from anywhere. Either alone is fine until somebody edits the other one.
Connect a data source
Data sources live under Connections in the left-side menu, where Add new connection is the button. Older writeups point at a top-level Configuration menu, which Grafana removed, so a screenshot that does not match your screen is probably from before that change.
Prometheus and the data source URL field
Prometheus is the default first data source. It scrapes, it stores and it answers PromQL over HTTP, which is exactly the second row of the table above. Install it with Node Exporter on the same host, point Grafana at http://localhost:9090, and you have infrastructure metrics in about twenty minutes.
The URL field is where an hour goes. Grafana's backend makes the query, not your browser, so the address has to resolve from wherever Grafana itself is running, and that stops being your shell the moment a container is involved. Containers sharing a network want the service name, http://prometheus:9090. A save and test that comes back with connection refused, on a port you can reach yourself, is this almost every time.
While you are on that page, set Scrape interval under Interval behavior so that it agrees with scrape_interval in prometheus.yml. Grafana derives $__rate_interval from that number, and when the two disagree your rate calculations come out quietly wrong rather than visibly broken.
For the full path from an empty Ubuntu box, connecting Grafana to Prometheus and building a dashboard goes through prometheus.yml, the systemd units, the community dashboard import and hand-written PromQL. Application metrics work the same way, with the application exposing its own /metrics endpoint and Prometheus scraping it alongside the host, and there is a worked example of that shape in monitoring a Node.js app with Prometheus and Grafana. The pattern transfers to anything that speaks the Prometheus exposition format.
SQL, Loki and Elasticsearch
Prometheus is not a prerequisite and plenty of Grafana instances never touch it.
SQL data sources get less credit than they deserve. A read-only PostgreSQL or MySQL user, ordinary SQL, the $__timeFilter(ts) macro dropped in where the WHERE clause goes: that is a business dashboard, built without an exporter or a metrics pipeline anywhere near it. Signups per day, failed payments and queue depth, read straight out of the application database.
For logs, pick between Loki and Elasticsearch on how each one indexes. Elasticsearch tokenises your document contents as they arrive, which buys fast search over terabytes and costs disk plus JVM heap. Loki keeps an index of labels only, compresses the chunks and parks them in object storage, so it is cheap to run and slow to answer a query whose label selector is vague. Daily log searching points at Elasticsearch. Logs you open after an alert fires point at Loki, and on a small server the memory gap between the two is enormous.
Dashboards and panels
A dashboard is panels plus a shared time range plus template variables. Template variables turn one dashboard into fifty: define a variable that queries the label values of instance, put $instance in every panel query, and you get a dropdown that repoints the whole page at a different host. Building four near-identical dashboards for four servers is the sign that nobody found this feature. Building and managing Grafana dashboards gives variable types, repeats, folders and the JSON model proper treatment, which this section does not. Read it once you have one panel drawing something.
On importing against building, do both and keep them for different purposes. Import dashboard 1860, Node Exporter Full, on day one. It is excellent reference material, every panel exposes its query, so reading it is a decent way into PromQL, and when something is misbehaving at 2am it already has disk latency broken out by device without you writing four queries.
Community dashboards are reference material and they are poor at being the screen you leave open. A wall of panels gets ignored, and one of them is always faintly red. Whatever you end up watching daily should be small enough that you wrote every query on it, which in practice means building it yourself over an afternoon.
Panel reducers versus the query
A panel reducer collapses a series into one value using functions with names like Last, Mean, Difference, Delta and Range. Those run inside Grafana, over a series already thinned down to the pixel width of the panel. The query runs inside your data source, against samples at full resolution. More wrong numbers come out of that distinction than out of any other part of Grafana.
A Stat panel asked for the change over the last week using the Difference reducer is subtracting the first point Grafana received from the last point Grafana received, and over a long window on a cramped panel that first point can sit hours from where your time picker starts. Ask the data source instead, with increase(http_requests_total[$__range]), and you get an answer that survives being quoted to somebody.
The reducer names cause their own trouble. Grafana's Delta only counts increments, so it is not the mathematical delta and it never goes negative. The full comparison of the reducers is in making a Grafana Stat panel show the delta over a time range, with the PromQL and SQL equivalents for each. Read it before you put a number on a screen somebody will act on.
Compute in the query, display in the panel. Anything you would defend in a meeting belongs on the data source side of that line, because the panel side quietly depends on how wide the browser window was.
Set up alerting
Get the mail path working first, then build the alerting objects. Doing it the other way round is why so many people end up with a rule going red on a dashboard while no email ever arrives.
The SMTP prerequisite
Self-hosted Grafana ships with email switched off. A minimal working block:
[smtp]
enabled = true
host = smtp.example.com:587
user = [email protected]
password = """your-password-here"""
from_address = [email protected]
from_name = Grafana
startTLS_policy = MandatoryStartTLSTwo things about that block. Port 587 under STARTTLS, or 465 with implicit TLS, against an authenticated relay is what you want, since outbound 25 is closed on most hosting networks and the shipped localhost:25 default is therefore going nowhere. And the triple quotes around the password are not decoration: a generated relay password containing # or ; parses as an empty value followed by a comment without them.
Which file that block belongs in depends entirely on how Grafana got onto the machine. Where to set up Grafana SMTP settings lists the per-install-type paths and the relay settings for the usual providers. Package installs, containers and the tarball each read a different file.
When your first test send fails with SMTP not configured, check your grafana.ini config file's [smtp] section, stop checking your host and port. That string is defined once in the Grafana source and returned by one guard on the enabled flag, before any hostname is resolved and long before a socket opens. It means the running process is holding a configuration where enabled reads false, and nothing else. Nine times out of ten the line in your file still carries the leading semicolon it shipped with, because the /etc/grafana/grafana.ini the package installs is a copy of sample.ini where every single line is commented out. The fix for the Grafana SMTP not configured error works through the other reasons it stays false, including a stale environment variable overriding a perfectly correct file.
Why the package ships that file fully commented rather than as a small file with live defaults, I do not know. Presumably so the shipped defaults stay in exactly one place and your file is a pure override layer. It costs people an hour each, reliably, and I have never seen a good argument for changing it either.
Contact points, rules and notification policies
Four objects have to line up before mail arrives. A contact point is a list of recipients. An alert rule pairs a query with a condition and an evaluation schedule. An evaluation group is a named bucket of rules sharing an interval. A notification policy reads the labels on a firing alert and decides which contact point hears about it.
Noise is governed by the pending period rather than by the threshold. Make a condition hold across five evaluations before Grafana calls the alert Firing and the transient spikes stop reaching anybody's inbox. Zero is the value to avoid.
The other half of that fix belongs in the query. Smooth a noisy gauge with avg_over_time() and wrap counters in rate() before the condition ever sees them.
Test the contact point before you write a single rule. Grafana's email contact point documentation covers the fields. For the complete flow from contact point through rule to routing, including the label matching that makes policies useful, setting up email alerts in Grafana builds one end to end.
Grafana alerting or Prometheus with Alertmanager
For a single VPS, keep alerting in Grafana. Rules, routing, silences and contact points live in one place with one UI, the rules can combine queries from several data sources in a way Prometheus structurally cannot, and Alertmanager becomes a second daemon to patch for no gain you can point at.
Move rules into Prometheus with Alertmanager delivering when one of two things becomes true. Either you need alerts to keep firing while Grafana is being restarted or upgraded, or you have several Prometheus servers whose alerts need shared deduplication, grouping and inhibition.
What I avoid in every case is running the same alert family in both systems. You get duplicate pages, and you inevitably fix the rule in the one that is not firing.
Self-hosted alerting has a circular problem sitting at the bottom of it. Mail server dies, Grafana tells you by mail. Give your highest severity a second contact point that does not touch SMTP at all.
Operating Grafana: logs, errors and upgrades
Where Grafana writes its log
On a deb or rpm install the log is /var/log/grafana/grafana.log, and journald has the same content because console mode is on by default. In a container the file exists and stays empty, and everything is in docker logs. On a tarball install it is data/log/grafana.log under the install directory.
Leave level at info and reach for filters in the [log] section, which lifts a single named logger while the rest of them stay where they are. Global debug on a live instance is unreadable:
[log]
level = info
filters = notifications:debug,ngalert:debugThat pair is for chasing an alert email that never arrived. sqlstore:debug covers database access. provisioning:debug covers dashboards and data sources arriving from files. plugin.loader:debug covers a plugin that installed and never appeared. The log paths per install type, the rotation settings and the reason a container's log file stays empty are in Grafana log file locations and log levels. Turn one filter on, reproduce the problem once, turn it back off.
Configuration is assembled in five passes, and the order settles most arguments about why a setting did nothing. Lowest first: conf/defaults.ini, the cfg:default.* arguments, your own file, the GF_* variables, then bare cfg:* arguments. The Docker entrypoint and the packaged systemd unit both write into layer two, which is why your file overrides them and why a stray environment variable overrides your file. Every key in every section is listed in the Grafana configuration reference.
Common Grafana errors
Start here, because it needs no authentication and it takes a second:
curl -s http://127.0.0.1:3000/api/healthA response carrying "database": "ok" tells you the process is alive and talking to its store, which leaves the browser, the proxy, one data source or the dashboard JSON. Put that endpoint in your uptime checks. A login page renders perfectly well with an unreachable database behind it.
The error strings themselves are precise if you know where they come from and vague if you do not. A process that starts and exits within a second is nearly always ownership of the data or log directory, since the service runs as the grafana user. A blank page reporting that Grafana failed to load its application files is root_url or serve_from_sub_path disagreeing with your reverse proxy, and the backend is healthy the whole time. A growing count of database is locked in the log means alerting has outgrown SQLite. A panel reading No data while the same query works in Prometheus is usually the time range, the step interval or a hardcoded range window shorter than your scrape interval. Each of those symptoms is mapped to its real cause in common Grafana errors and how to fix them, with the first command worth running for each.
Grafana security settings to change
Set secret_key in [security] before the instance holds anything. Generate one with openssl rand -hex 32.
Data source credentials and notification settings are encrypted in the database with that key. Leave it unset and the fallback is a constant shipped in the public source tree, the same one on every Grafana anywhere. Change it after secrets exist and those secrets stop decrypting until you run a re-encryption, which is why this belongs at the start.
After that: bind to loopback and terminate TLS in a real web server, set cookie_secure = true once TLS is in front, leave brute force protection on and give anything automated a service account rather than a copy of the admin password. Grafana security best practices and hardening carries the whole checklist with the reasoning attached to each item. The shipped defaults are tuned to get a first dashboard drawing quickly, and about a dozen one-line changes move an instance from that state to something you can leave facing the internet.
Upgrades and the support window
Nine months of support per minor version, fifteen for the final minor of a major line. New minors arrive on a two month cadence with patch releases between them. The current release is 13.2.0, published on 18 August 2026 and supported until 18 May 2027, with 13.1.x, 13.0.x and 12.4.x also still inside their windows.
A year without attention puts an instance outside every supported window. Upgrading a package install is one command, and the service bounces itself afterwards because /etc/default/grafana-server carries RESTART_ON_UPGRADE=true:
sudo apt-get update && sudo apt-get install --only-upgrade grafanaBack up /var/lib/grafana/grafana.db first. Grafana migrates its schema forward at startup and never migrates back, so that file is your only route out of a bad upgrade, and a downgrade against an already-migrated schema is not a supported operation. Grafana's guidance on when to upgrade lists which versions are still supported. Check it before deciding a version is fine on the grounds that it works.
Automate Grafana with provisioning, the API and MCP
Provisioning files, the HTTP API and the MCP server all cut down on clicking, and they are not interchangeable.
Provisioning is YAML that Grafana reads out of /etc/grafana/provisioning when it starts. A data source written that way applies cleanly however many times it runs, ships in the same repository as everything else you deploy, comes back on its own after a rebuild and can be marked editable: false so the UI cannot drift away from the file. Anything a human should review in a pull request belongs here.
The HTTP API is for state rather than configuration. Backing up dashboards people built in the UI, reading which alerts fired last night, creating a folder the moment a new customer signs up. Authentication is a service account token in an Authorization: Bearer header, since API keys were retired in favour of service accounts, and because a single account can carry more than one token at a time, rotation needs no window of downtime. The curl recipes, the newer /apis route structure and a working dashboard backup script are in the Grafana API and service accounts guide. Note that server-admin endpoints do not accept tokens at all and want basic auth as a Grafana server administrator, which sits behind most confusing 401s.
The newest option is the Model Context Protocol server, mcp-grafana, which exposes your instance as callable tools so an assistant can search dashboards, run PromQL, read alert state and answer a question by going and looking rather than reciting how alerting works in general. It authenticates with the same service account token.
Scope is the thing to think about before you run it. A token that reads every dashboard is a fairly complete map of your infrastructure, because dashboard queries carry hostnames, database names, namespaces and customer identifiers in label values. Run it with a Viewer service account and the --disable-write flag, which are two independent things saying no. Installation, transports and the permission model are in the setup guide for the Grafana MCP server.
Grafana compared to Datadog, Kibana and lighter tools
Grafana against a commercial platform is a question about money shape. Datadog's invoice grows with the estate and asks almost nothing of your people. Self-hosting keeps the invoice flat and puts the hours on somebody in your team. Where Datadog wins outright is application performance monitoring: instrumentation that attaches itself, service dependency maps and a click from a slow trace to whatever the host was doing in that same second, none of which you have to build. Grafana versus Datadog has the full comparison, including the custom metric mechanism that turns a predictable bill into a surprising one. That correlation is the hardest thing to assemble yourself and it is worth being honest about.
Grafana against Kibana is a question about what your primary data is. Kibana is the front end for Elasticsearch and it talks to nothing else, which buys real depth on that one backend: Discover, field statistics, KQL and ES|QL. Grafana reads twenty backends shallowly. If log search is somebody's daily job, take Kibana and give Elasticsearch the RAM it needs. If your data is spread across several systems and you want one time picker over all of it, that is the case Grafana was built for, and Grafana versus Kibana covers the details including the free-tier notification limits.
A third possibility gets considered less often than it should: you picked something too heavy. A real share of Grafana installations exist only to alert when a website stops responding, which a small uptime monitor does in five minutes without a data source or a query language anywhere in the picture. Another share are business intelligence tools wearing a monitoring costume, drawing signups and refunds out of Postgres, where a proper BI product would let the finance team build their own charts unaided. The Grafana alternatives roundup sorts ten tools by the problem each one really solves, which is the useful way to shop.
What Grafana is weak at
Two things, and being specific about them is more useful than a caveat paragraph.
The first is dashboard lifecycle. Grafana makes creating a dashboard cheap and gives you almost nothing for retiring one. Open source Grafana has no usage statistics, no last-viewed column and no ownership field, so an instance that has been running for three years holds hundreds of dashboards and nobody can say which fifteen are load-bearing. The workaround is discipline rather than a feature: folders that map to teams, provisioning the dashboards that matter from files so the Git history is the ownership record, and a willingness to delete anything nobody objects to. It is a real gap and no amount of configuration closes it.
The second is that Grafana has no opinion about what should be measured. It draws what you ask for and it will happily render a beautiful panel for an exporter that died an hour ago, because a query returning nothing looks the same as a metric that is genuinely at zero. Nothing in the product notices that a target stopped reporting unless you write the rule that notices, which is why the no-data handling on an alert rule deserves more thought than the threshold does. Products that own collection as well as display, Netdata and Zabbix among them, know when a host went quiet because they were the ones asking.
Neither of those makes Grafana the wrong choice. They are the two places where the read-layer design has a cost, and knowing where the cost falls is the difference between a monitoring setup you trust and one you look at.
If you take one operational habit from this page, take the config precedence order. Run sudo journalctl -u grafana-server -n 200 --no-pager | grep -E 'Config loaded|Config overridden' after any change and Grafana tells you, line by line, which layer won. That output answers more Grafana questions than any config file will.
