.svg)
.svg)
.svg)
.svg)
.svg)
.svg)
.svg)
Prometheus goes on the box first, then Node Exporter, then a data source entry in Grafana pointing at Prometheus. Prometheus does the scraping and the storage. Node Exporter turns a Linux host into something scrapeable. Grafana holds no metrics of its own and queries Prometheus fresh every time a panel loads.
All of it fits on a single Ubuntu VPS. Install Prometheus from the official tarball, install Node Exporter, connect Grafana to Prometheus through the Grafana data sources page, import the community dashboard everyone uses, then build one panel by hand. That last step is the one to keep. Importing a dashboard teaches you nothing about the query editor, and the query editor is where you will spend every hour after the first one.
Before you start
You need a running Grafana. If you do not have one yet, installing Grafana on an Ubuntu VPS from the APT repository takes a few minutes and gives you the packaged layout that the rest of this assumes. Everything below was run on Ubuntu 24.04 with Grafana 13.x, and none of it is version-sensitive enough to break on 26.04.
Two versions to note, because you will be typing them into URLs: Prometheus 3.14.0 and Node Exporter 1.12.1 are current as of writing. Check the releases pages and substitute if newer ones have landed. The layout of the commands does not change.
Install Prometheus on Ubuntu
Create a system user and directories
Prometheus should not run as root and it should not run as your login user either. Give it an account with no shell and no home directory:
sudo useradd --no-create-home --shell /usr/sbin/nologin prometheus
sudo mkdir -p /etc/prometheus /var/lib/prometheus/etc/prometheus holds the config. /var/lib/prometheus holds the time series database. Keeping them apart means you can back up the first and ignore the second.
Download and unpack the binaries
cd /tmp
wget https://github.com/prometheus/prometheus/releases/download/v3.14.0/prometheus-3.14.0.linux-amd64.tar.gz
tar xzf prometheus-3.14.0.linux-amd64.tar.gz
cd prometheus-3.14.0.linux-amd64
sudo cp prometheus promtool /usr/local/bin/
sudo cp prometheus.yml /etc/prometheus/
sudo chown prometheus:prometheus /usr/local/bin/prometheus /usr/local/bin/promtool
sudo chown -R prometheus:prometheus /etc/prometheus /var/lib/prometheusThe 3.x archive contains five things: LICENSE, NOTICE, prometheus.yml, promtool and prometheus. There is no consoles/ directory and no console_libraries/. If a guide tells you to copy those and pass --web.console.templates, that guide predates Prometheus 3 and the cp will fail before you even get to the flags. Skip those lines entirely.
Write prometheus.yml
The shipped config scrapes only Prometheus itself. Replace it with something that also knows about Node Exporter:
sudo nano /etc/prometheus/prometheus.ymlglobal:
scrape_interval: 15s
evaluation_interval: 15s
scrape_configs:
- job_name: "prometheus"
static_configs:
- targets: ["localhost:9090"]
- job_name: "node"
static_configs:
- targets: ["localhost:9100"]
labels:
instance: "web-01"The instance label is optional and I set it on every job I write. Without it you get localhost:9100 as the instance name in every dashboard legend, which stops being readable somewhere around the third server. Set it once here and every panel inherits it.
Check the file before you start anything, because YAML indentation errors produce a service that exits immediately with a message you will only find in the journal:
promtool check config /etc/prometheus/prometheus.ymlCreate the systemd unit
Prometheus ships no unit file, so write one at /etc/systemd/system/prometheus.service:
[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target
[Service]
User=prometheus
Group=prometheus
Type=simple
Restart=on-failure
ExecStart=/usr/local/bin/prometheus \
--config.file=/etc/prometheus/prometheus.yml \
--storage.tsdb.path=/var/lib/prometheus \
--web.listen-address=127.0.0.1:9090
[Install]
WantedBy=multi-user.targetsudo systemctl daemon-reload
sudo systemctl enable --now prometheus
sudo systemctl status prometheusThat --web.listen-address=127.0.0.1:9090 is deliberate. The default is 0.0.0.0:9090, so a fresh Prometheus with no firewall in front of it is readable by the entire internet. Binding it to loopback on the first boot means the window never opens.
Tunnel the Prometheus web UI when you want to look at it from your laptop:
ssh -L 9090:127.0.0.1:9090 youruser@your-serverInstall Node Exporter
Node Exporter is the piece that reads /proc and /sys and exposes CPU, memory, disk and network counters on port 9100. Same shape of install:
sudo useradd --no-create-home --shell /usr/sbin/nologin node_exporter
cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.12.1/node_exporter-1.12.1.linux-amd64.tar.gz
tar xzf node_exporter-1.12.1.linux-amd64.tar.gz
sudo cp node_exporter-1.12.1.linux-amd64/node_exporter /usr/local/bin/
sudo chown node_exporter:node_exporter /usr/local/bin/node_exporterThe unit goes in /etc/systemd/system/node_exporter.service:
[Unit]
Description=Prometheus Node Exporter
Wants=network-online.target
After=network-online.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
Restart=on-failure
ExecStart=/usr/local/bin/node_exporter --web.listen-address=127.0.0.1:9100
[Install]
WantedBy=multi-user.targetsudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
curl -s http://127.0.0.1:9100/metrics | grep node_cpu_seconds_total | head -5If that last command prints counter lines, the exporter works. Prometheus will pick it up on the next scrape, so wait fifteen seconds and check the targets page (through the tunnel, at http://localhost:9090/targets). A job in state UP means the whole chain from exporter to storage is intact. Anything that goes wrong past that point is on the Grafana side.
This only applies if you intend to run the Node Exporter Full dashboard as shipped. It suggests enabling --collector.systemd and --collector.processes, which populates a handful of panels that would otherwise sit empty. Both add a small amount of scrape cost. I turn on systemd and leave processes off unless I am chasing something specific.
Ports and paths so far. Prometheus listens on 9090 and Node Exporter on 9100, both bound to loopback. The Prometheus config is /etc/prometheus/prometheus.yml and its data sits under /var/lib/prometheus. Both binaries are in /usr/local/bin. The units are /etc/systemd/system/prometheus.service and /etc/systemd/system/node_exporter.service. Grafana is on 3000 wherever your install put it.
Add Prometheus as a Grafana data source
In the Grafana UI, click Connections in the left-side menu, then Add new connection. Type Prometheus into the search bar, click Prometheus data source, then Add new data source in the upper right. That path changed a while back and half the tutorials on the web still send you to Configuration, which no longer exists as a top-level menu item.
What to put in the URL field
The Grafana docs describe the Connection URL as "The URL of your Prometheus server. If Prometheus is running locally, use http://localhost:9090." That works for the setup above, where both processes run on the same host as ordinary services.
The moment either side moves into a container, that changes. If Grafana runs in Docker and Prometheus runs on the host, localhost inside the Grafana container means the Grafana container. If both run in Docker on the same user-defined network, use the container name: http://prometheus:9090. The Grafana Docker Compose setup covers the networking side in more detail, and a save-and-test that returns connection refused while the port is demonstrably open is nearly always this. It catches people who otherwise know exactly what they are doing.
While you are on this page, find Scrape interval under Interval behavior and set it to match the scrape_interval in your prometheus.yml. In this build that is 15s. Grafana uses that number to compute $__rate_interval, and if the two disagree your rate calculations quietly come out wrong. Then click Save & test.
Import the Node Exporter Full dashboard
Dashboard ID 1860, called Node Exporter Full, is the one everybody uses and it is still actively maintained. Its own notes say revision 16 onward targets prometheus-node-exporter v0.18 or newer, so a current exporter is well inside the supported range.
Import it from Dashboards, using New and then Import dashboard. Paste 1860 in, hit Load, choose your Prometheus data source from the dropdown and confirm with Import.
You now have dozens of panels covering memory pressure, filesystem fill, disk latency, network throughput and a lot else. Some will show No data. That is normal and it is almost always a collector you have not enabled. The guide to Grafana dashboards covers trimming an imported dashboard down to the panels you will read and pointing it at a variable instead of one hard-coded host.
Build a CPU panel by hand
Create a new dashboard, click Add visualization and pick your Prometheus data source. The query editor opens in Builder mode by default. Switch it to Code. Builder mode is useful for exploring the label values on a metric you have never seen before. After that it slows you down.
Type this:
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[$__rate_interval])) * 100)Reading it from the inside out: node_cpu_seconds_total is a counter of seconds each CPU has spent in each mode, rate() converts that to seconds-per-second (which is a fraction of one core), {mode="idle"} keeps only idle time, avg by (instance) collapses the per-core series into one number per host and subtracting from 100 flips idle into busy.
There is a reason everyone subtracts idle instead of adding up the busy modes. The list of modes has grown over the years and varies by kernel, so a sum over user, system, nice, irq, softirq and steal is a list you would have to keep maintaining. Idle is one label and it has never moved. Back to the panel.
Set the panel unit to Percent (0-100) under Standard options, give it a title, and you have a CPU panel that is honest on any core count.
The version the official node-mixin uses is stricter, because it also excludes iowait and steal from the idle bucket:
1 - avg without (cpu) (
sum without (mode) (rate(node_cpu_seconds_total{mode=~"idle|iowait|steal"}[$__rate_interval]))
)That one returns a ratio between 0 and 1, so set the unit to Percent (0.0-1.0) instead. On a VPS the steal figure deserves a panel of its own, since it tells you how long your vCPU sat ready to run while the hypervisor was busy elsewhere. Sustained steal above a few percent is a host contention problem. It is one of the few numbers on the dashboard that describes the machine underneath you.
PromQL expressions for memory, disk and network
| What it shows | Expression |
|---|---|
| Memory used, as a ratio | 1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) |
| Root filesystem used, percent | 100 - (node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"} * 100) |
| Inbound network, bits per second | rate(node_network_receive_bytes_total{device!="lo"}[$__rate_interval]) * 8 |
| Outbound network, bits per second | rate(node_network_transmit_bytes_total{device!="lo"}[$__rate_interval]) * 8 |
| Disk busy time, ratio | rate(node_disk_io_time_seconds_total[$__rate_interval]) |
| Load average per core | node_load1 / count without (cpu, mode) (node_cpu_seconds_total{mode="idle"}) |
A few notes on those. Use MemAvailable rather than MemFree: free memory on a healthy Linux box is close to zero because the kernel uses everything spare for page cache, and a panel built on MemFree will have you paging yourself at 3am over a machine that is perfectly fine. The filesystem query wants a fstype filter too if you have overlay or squashfs mounts, something like fstype!~"tmpfs|squashfs|overlay", otherwise every snap package on the box shows up as a full disk. And the * 8 on the network queries converts bytes to bits, which is what you want if you are comparing against a port speed quoted in Gbps.
The disk expression returns a value near 1.0 when the device is saturated. Set the unit to Percent (0.0-1.0) and it reads like the busy column in iostat.
rate, irate and $__rate_interval
rate() calculates the per-second average rate of increase across the whole range you give it, and it corrects for counter resets when a process restarts. irate() calculates an instant rate from only the last two data points in the range. The Prometheus function reference is blunt about the difference: "irate should only be used when graphing volatile, fast-moving counters. Use rate for alerts and slow-moving counters."
In practice I use rate() for everything on a dashboard. irate() produces spikier graphs that look more responsive and hide real behaviour, because on a wide time range Grafana asks for one point every few minutes and irate() throws away every sample except the final pair in each of those windows. You end up graphing noise.
Then there is the range window. Writing rate(metric[5m]) with a hardcoded window works until someone zooms out to thirty days, at which point Grafana is stepping every ten minutes and a five-minute window contains no complete pair of samples. The graph goes blank in patches. Writing rate(metric[$__interval]) is worse, because at tight zoom levels $__interval can drop below the scrape interval and give you empty windows.
$__rate_interval exists to fix exactly this. Grafana computes it as max($__interval + scrape interval, 4 * scrape interval), which guarantees the window always contains at least four samples no matter how far in or out you zoom. The scrape interval in that formula comes from the per-query Min step if you set one, otherwise from the Scrape interval field on the data source, which is why the earlier step of matching it to prometheus.yml matters. The Grafana Prometheus variables documentation puts it plainly: always use $__rate_interval instead of a fixed interval or $__interval with rate() and increase().
Community dashboards versus your own
Import 1860 on day one and keep it. Do not put it on a wall.
Community dashboards are excellent reference material. When something is misbehaving and you want to see disk latency broken out by device without writing four queries, 1860 is already there and already correct. It is also a good way to learn PromQL by reading, since every panel exposes its query and most of them are better written than what you would produce cold.
As a monitoring dashboard it fails. Dozens of panels means nobody looks at any of them, and a screen where everything is always slightly red trains people to ignore red. The dashboard you keep open should have six to eight panels covering the things that would make you get up: CPU, memory, root filesystem, the error rate of the thing you run and one latency number. Build that one yourself. It takes an hour and you will understand every query on it.
Applications work the same way. The Node.js monitoring with Prometheus and Grafana walkthrough hangs a handful of app-specific panels off the Prometheus you just built, so there is no second stack to stand up.
Prometheus authentication and binding to localhost
Prometheus has no user database, no login screen and no concept of a session. Anyone who can reach port 9090 can read every metric you have ever collected, run arbitrary PromQL against it and read your entire config from /api/v1/status/config, which includes any scrape credentials you put there in plain text. TLS and basic auth are available through --web.config.file, and that flag is still marked experimental in the Prometheus command-line reference, which makes it a thin thing to build an access control model on.
The answer is already baked into the unit above. Bind Prometheus to 127.0.0.1, bind Node Exporter to 127.0.0.1, and let Grafana reach both over loopback. Grafana is then the only thing listening on a public interface, and Grafana does have accounts, roles and a login you can put behind TLS. If a reverse proxy is going in front of it, a change to the Grafana port belongs in the same afternoon, and the Grafana security settings worth changing cover the rest of that surface.
Two flags to leave alone while you are here. --web.enable-lifecycle exposes a /-/reload endpoint so you can reload config without restarting, handy on a config-managed fleet and an unauthenticated restart button on a box reachable from outside. --web.enable-admin-api exposes endpoints that can delete series. Both default to false. Both should stay there unless you have a specific reason and a private network to run them on.
If you would rather not do the install by hand at all, the one-click Grafana VPS and Prometheus VPS templates deploy either one already running, and you can put both on the same instance and skip straight to the data source step.
One last thing before you close the tab. Go back to /targets and note the Last Scrape column. If it ever reads longer than your scrape interval, Prometheus is falling behind, and on a small VPS that is nearly always Node Exporter's filesystem collector stalling on a network mount. Why it stalls for the whole scrape instead of failing on the one mount, I have not worked out.
