Grafana: Installation and Setup, Dashboards, Prometheus and Zabbix

Grafana: Installation and Setup, Dashboards, Prometheus and Zabbix
Quick answer
Grafana doesn’t collect or store anything on its own. It’s a storefront: it draws graphs from data sitting in Prometheus, Zabbix, PostgreSQL and other sources. A working setup comes down to four steps:

Empty panels with “No data” almost always mean the wrong data source address or the wrong time range selected.

Installing Grafana on Ubuntu 24.04 from the official repository takes five commands:


# On grafana-01 (Ubuntu 24.04): install Grafana OSS from the Grafana Labs repository
sudo apt-get install -y apt-transport-https wget gnupg
sudo mkdir -p /etc/apt/keyrings
sudo wget -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update && sudo apt-get install -y grafana && sudo systemctl enable --now grafana-server

Diagnosis: why Grafana shows nothing after installation

You installed Grafana, opened port 3000, logged in. And it’s empty: no graphs, no metrics, not a single server. Sound familiar?

Here’s the thing. Grafana isn’t a monitoring system in the pure sense. It’s a visualization layer. It doesn’t poll servers, doesn’t store history, and knows nothing about your infrastructure until you connect a data source to it. Metrics live in Prometheus, in Zabbix, in PostgreSQL, in InfluxDB. Grafana queries them and draws the response.

This is where the main beginner mistake comes from: installing Grafana and expecting it to “show something on its own.” It won’t. A Grafana-based monitoring system is always a chain: collector plus storage plus dashboard.

What you’ll get from this article: a working Grafana 13 set up two ways, on packages and in Docker Compose. A Prometheus and node_exporter stack for Linux metrics. Zabbix connected through the official plugin, so you can see Zabbix problems and graphs on one dashboard. HTTPS through nginx, Grafana running on PostgreSQL instead of SQLite, backup, upgrades and error troubleshooting.

Time: 40-60 minutes total. If you only need Grafana with Zabbix, you’ll fit it in 20.

What you’ll need:

  • A server or VM, grafana-01, with Ubuntu 24.04, address 10.10.0.20, 2 vCPU and 2-4 GB of RAM.
  • Root or sudo on that machine.
  • For the Zabbix integration: a working Zabbix server 7.0 on zbx-server, 10.10.0.10, and admin rights in its web interface.
  • A domain or internal DNS name for HTTPS, for example grafana.example.local.

What’s covered in this article:

  • Architecture: who polls whom and where the data lives.
  • Seven reasons panels show No data.
  • Installing Grafana on packages and in Docker Compose along with Prometheus.
  • The Zabbix plugin for Grafana: setup with an API token and ready-made dashboards.
  • Your own dashboard with variables, nginx with HTTPS, PostgreSQL as the Grafana database, working with the API.
  • Verification, errors with commands, Grafana alternatives.
  • Monitoring Grafana itself, backup, upgrades and FAQ.

Architecture: where Grafana is and where the data is

Look at this diagram before you install anything. Half the “why isn’t this working” questions go away once you understand who talks to whom.

%%{init: {
  'theme': 'base',
  'themeVariables': {
    'primaryColor': '#ffffff',
    'primaryTextColor': '#1e293b',
    'primaryBorderColor': '#94a3b8',
    'lineColor': '#64748b',
    'fontSize': '15px',
    'fontFamily': 'ui-sans-serif, system-ui, sans-serif'
  },
  'flowchart': {'curve': 'linear', 'nodeSpacing': 50, 'rankSpacing': 50}
}}%%
flowchart TD
    U["User's browser"] --> N["nginx with HTTPS"]
    N --> G["Grafana"]
    G --> GDB["Grafana database: dashboards and users"]
    G --> P["Prometheus"]
    G --> Z["Zabbix API"]
    G --> S["PostgreSQL as a data source"]
    NE["node_exporter on servers"] --> P
    ZA["Zabbix agents"] --> ZS["Zabbix server"]
    ZS --> Z
    style U fill:#f8fafc,stroke:#3b82f6,stroke-width:2px,color:#1e40af
    style N fill:#f8fafc,stroke:#3b82f6,stroke-width:2px,color:#1e40af
    style G fill:#f8fafc,stroke:#22c55e,stroke-width:2px,color:#15803d
    style GDB fill:#f8fafc,stroke:#22c55e,stroke-width:2px,color:#15803d
    style P fill:#f8fafc,stroke:#3b82f6,stroke-width:2px,color:#1e40af
    style Z fill:#f8fafc,stroke:#3b82f6,stroke-width:2px,color:#1e40af
    style S fill:#f8fafc,stroke:#3b82f6,stroke-width:2px,color:#1e40af
    style NE fill:#f8fafc,stroke:#3b82f6,stroke-width:2px,color:#1e40af
    style ZA fill:#f8fafc,stroke:#3b82f6,stroke-width:2px,color:#1e40af
    style ZS fill:#f8fafc,stroke:#3b82f6,stroke-width:2px,color:#1e40af

Here’s the key point: Grafana has its own database, but it only holds dashboards, users, data source settings and alert rules. No metrics are stored there. By default it’s an SQLite file at /var/lib/grafana/grafana.db. Lose that file and you lose all dashboards, but not a single data point on the graphs.

Every time you open a dashboard, here’s what happens:

%%{init: {
  'theme': 'base',
  'themeVariables': {
    'primaryColor': '#f8fafc',
    'primaryTextColor': '#1e293b',
    'primaryBorderColor': '#94a3b8',
    'noteBkgColor': '#fefce8',
    'noteTextColor': '#713f12',
    'noteBorderColor': '#fbbf24',
    'actorBkg': '#f8fafc',
    'actorBorder': '#94a3b8',
    'actorTextColor': '#1e293b',
    'fontSize': '15px',
    'fontFamily': 'ui-sans-serif, system-ui, sans-serif'
  },
  'sequence': {
    'mirrorActors': false,
    'messageAlign': 'center',
    'actorMargin': 120,
    'width': 160,
    'noteMargin': 12
  }
}}%%
sequenceDiagram
    participant B as Browser
    participant G as Grafana
    participant P as Prometheus
    B->>G: Open dashboard
    G->>P: PromQL query for a time range
    P->>G: Time series
    G->>B: Data for panels
    Note over G,P: The request comes from the Grafana server, not the browser

That last line in the diagram is the most important one. Grafana opens the data source address from its own server. If Grafana is in a container, localhost means the container itself, not the host. This is where every second person who runs Grafana in Docker trips up.

Causes: why there’s no data in Grafana

Before you install and configure anything, here’s a list of what breaks most often. Half the items are fixed by correct installation, the other half by understanding the diagram above.

# Cause Why it breaks
1 Data source isn’t added or added incorrectly Panels exist, but there’s no one to ask for data. An imported dashboard looks for a source that doesn’t exist
2 Source address set to localhost while Grafana runs in Docker The container knocks on its own door and gets connection refused
3 Wrong time range or time zone Data exists, just for a different period. Clock drift on the server shifts the points
4 Zabbix plugin installed but not enabled Without enabling the app, the Zabbix source type isn’t available
5 Zabbix user has no permissions on host groups The API honestly returns an empty host list, and Grafana draws nothing
6 Prometheus isn’t scraping the target or labels in the query are wrong The query is correct, but there are no matches. The target is in a down state
7 Wrong root_url behind a reverse proxy The UI loads partially, API requests go to the wrong place, you get endless redirects

The third cause deserves a separate mention. I once spent an hour figuring out why a colleague’s dashboard was empty while mine was full. Turned out he’d accidentally set the time range to “Last 5 minutes” in the top right corner, on a dashboard where the metric was collected every 15 minutes. Grafana was showing everything correctly. Just nothing.

The recipe: installing and setting up Grafana step by step

System requirements

At the time of publishing, Grafana 13.2.4 from October 13, 2026 is current. The 13 branch came out in April 2026, and 13.1, 13.0 and 12.4 still get security fixes in parallel. Before installing, check the latest releases on the Grafana download page and branch support timelines on endoflife.date.

Component Version at time of publishing Comment
OS Ubuntu 24.04 LTS, Debian 12, RHEL 9 Any modern 64-bit Linux
Grafana OSS 13.2.4 grafana package, grafana/grafana Docker image
Prometheus 3.13.4 LTS or 3.15.0 For production, take LTS
node_exporter 1.12.1 Installed on every monitored Linux server
Zabbix plugin 6.9.1 Requires Grafana 11.6 or newer
Zabbix server 7.0 LTS or 7.4 API token login needs Zabbix 5.4 or newer
Grafana database SQLite by default, PostgreSQL 14+ PostgreSQL for multiple instances and reliability

Check Prometheus and node_exporter versions on the Prometheus download page, and the Zabbix plugin on grafana-zabbix releases on GitHub.

Port table

Port Protocol Service Purpose Exposed externally?
443 TCP nginx HTTPS access for users to Grafana Yes, for users
80 TCP nginx Redirect to HTTPS and certificate issuance Yes, redirect only
3000 TCP Grafana Web UI and API No, localhost only behind nginx
9090 TCP Prometheus Query API and web UI No, only Grafana and admins
9100 TCP node_exporter Exposes server metrics No, only from the Prometheus address
443 or 80 TCP Zabbix frontend Zabbix API for the plugin Only from the Grafana address
5432 TCP PostgreSQL Grafana database or direct connection to the Zabbix database No

Packages or Docker: which to choose

Criterion apt/dnf packages Docker Compose
Time to first launch 5 minutes 5 minutes, if Docker is already installed
Prometheus alongside Installed separately Comes up in the same file
Upgrade apt upgrade Change the image tag
Troubleshooting at 3 a.m. One process, one log, systemd Plus container networking and volume permissions
When to choose A dedicated machine for Grafana The whole monitoring stack on one host, a lab, quick start

My choice for production is packages on a dedicated VM. For a test bench, home lab, or when you need the whole Grafana plus Prometheus stack at once, go with Docker Compose. Both paths are below, pick one.

Preparation: check your environment

Time on the server needs to be synced. Clock drift between Grafana, Prometheus and Zabbix produces shifted graphs and odd gaps in the data.


# On grafana-01: hostname, OS and time sync
hostnamectl
sudo apt-get update && sudo apt-get install -y chrony curl jq sqlite3
sudo systemctl enable --now chrony
timedatectl show -p NTPSynchronized

Result: NTPSynchronized=yes. If it’s no, check that UDP 123 to the NTP server isn’t being blocked.

If you’re planning to connect Zabbix, check right away that the Zabbix API is reachable from the Grafana machine. Grafana will be reaching it from exactly here.


# On grafana-01: does the Zabbix API respond
curl -s -X POST -H 'Content-Type: application/json-rpc' \
  -d '{"jsonrpc":"2.0","method":"apiinfo.version","params":{},"id":1}' \
  http://10.10.0.10/api_jsonrpc.php

Expected response: JSON with a result field and the Zabbix version, for example 7.0.25. If it’s empty or 404, the path to the API is different. In a package install of Zabbix on Apache it’s often /zabbix/api_jsonrpc.php.

Path A, step 1. Install Grafana from the repository

Why the official repository: distro repositories either don’t have Grafana or carry an outdated version. The Grafana Labs repository gives you current versions and security updates through regular apt upgrade.


# On grafana-01 (Ubuntu/Debian): Grafana Labs key and repository
sudo apt-get install -y apt-transport-https wget gnupg
sudo mkdir -p /etc/apt/keyrings
sudo wget -O /etc/apt/keyrings/grafana.asc https://apt.grafana.com/gpg-full.key
sudo chmod 644 /etc/apt/keyrings/grafana.asc
echo "deb [signed-by=/etc/apt/keyrings/grafana.asc] https://apt.grafana.com stable main" | sudo tee /etc/apt/sources.list.d/grafana.list
sudo apt-get update
sudo apt-get install -y grafana

The grafana package is the OSS edition. There’s also grafana-enterprise: free without a license, functionally equal to OSS, with paid features unlocked by a key. For internal monitoring, go with grafana, the AGPLv3 license is clearer here.

For RHEL 9, the repository is described by a file:


# On grafana-01 (RHEL 9): Grafana Labs repository and installation
sudo tee /etc/yum.repos.d/grafana.repo > /dev/null <<'EOF'
[grafana]
name=grafana
baseurl=https://rpm.grafana.com
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://rpm.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
EOF
sudo dnf install -y grafana

Result: the package is installed, /etc/grafana/grafana.ini and the grafana-server service show up.

If the download returns 451
From some addresses, Grafana Labs servers return a 451 geofence blocked error. This has been officially confirmed for Enterprise builds, but it shows up on other downloads too. Working workarounds: your own repository mirror in Nexus or Artifactory, downloading the package on a machine outside the block and transferring it over scp, or an image from a corporate Docker registry. The principle is the same: download once, keep it yourself, install from your own source.

# On a machine with repository access: download a specific .deb version without installing
apt-get download grafana=13.2.4
# On the same machine: transfer the package to the Grafana server
scp grafana_13.2.4_amd64.deb admin@10.10.0.20:/tmp/

# On grafana-01: install from the file
sudo apt-get install -y /tmp/grafana_13.2.4_amd64.deb

Path A, step 2. Basic grafana.ini configuration

Why: default settings are designed for “install and take a look.” Sign-up is open for invited users, Grafana listens on all interfaces, telemetry is sent out. For production, we close off the extras.

Don’t edit grafana.ini wholesale. Find the sections you need and uncomment the lines. All parameters can also be set as environment variables in the form GF_SECTION_KEY, which is handy in Docker.


# /etc/grafana/grafana.ini on grafana-01: what to change
[server]
http_addr = 127.0.0.1
http_port = 3000
domain = grafana.example.local
root_url = https://grafana.example.local/

[analytics]
reporting_enabled = false
check_for_updates = false
check_for_plugin_updates = false

[security]
cookie_secure = true
disable_gravatar = true

[users]
allow_sign_up = false
allow_org_create = false

[auth.anonymous]
enabled = false

Let’s go over the less obvious parts. http_addr = 127.0.0.1 makes Grafana listen only on the local interface: only nginx will be allowed in from the outside. Until nginx is set up, temporarily leave it empty, otherwise you won’t be able to log in from your workstation. root_url must match the address users actually open Grafana through. A mistake here causes redirects to localhost:3000 and broken links in notifications.

Only turn on cookie_secure = true once HTTPS is working. Over plain HTTP the browser won’t save that kind of cookie, and you won’t be able to log in.

Path A, step 3. Start and first login


# On grafana-01: start, enable autostart and check
sudo systemctl daemon-reload
sudo systemctl enable --now grafana-server
sudo systemctl status grafana-server --no-pager
curl -s http://127.0.0.1:3000/api/health | jq

Result: status active running, and /api/health returns database ok plus the version. Open the server’s address on port 3000 in your browser, login admin, password admin. Grafana will immediately ask you to change the password. Change it. Don’t click Skip.

The admin/admin password on an internal server that “nobody will find” is like leaving a spare key under the doormat. The first scanner that hits your network will find it, along with every credential saved in your data sources.

If you need to set the password before the first login, for example in a deployment script, use the reset command. In Grafana 13 the separate grafana-cli binary no longer works, use the grafana cli subcommand instead:


# On grafana-01: set the admin password from the console
sudo grafana cli --homepath /usr/share/grafana --config /etc/grafana/grafana.ini admin reset-admin-password 'NewStrongPassword'

Path B. Grafana and Prometheus in Docker Compose

Why: bring up the whole stack with one file: Grafana, Prometheus and node_exporter, with a data source already connected. In five minutes you’ll have graphs for the host everything’s running on.

About the images. As of the 12.4 branch, the grafana/grafana-oss repository is no longer updated. Use grafana/grafana, that’s the OSS build. Always pin the tag to a specific version: latest in production is a lottery with every docker compose pull.


# On grafana-01: directory structure for the stack
sudo mkdir -p /opt/monitoring/{prometheus,grafana/provisioning/{datasources,dashboards,plugins},grafana/dashboards}
cd /opt/monitoring

# On grafana-01: docker-compose.yml for the Grafana + Prometheus + node_exporter stack
sudo tee /opt/monitoring/docker-compose.yml > /dev/null <<'EOF'
services:
  grafana:
    image: grafana/grafana:13.2.4
    container_name: grafana
    restart: unless-stopped
    ports:
      - "127.0.0.1:3000:3000"
    environment:
      GF_SECURITY_ADMIN_USER: admin
      GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_admin
      GF_USERS_ALLOW_SIGN_UP: "false"
      GF_ANALYTICS_REPORTING_ENABLED: "false"
      GF_SERVER_ROOT_URL: https://grafana.example.local/
      GF_PLUGINS_PREINSTALL: alexanderzobnin-zabbix-app@6.9.1
      ZABBIX_API_TOKEN: ${ZABBIX_API_TOKEN:-}
    secrets:
      - grafana_admin
    volumes:
      - grafana-data:/var/lib/grafana
      - ./grafana/provisioning:/etc/grafana/provisioning:ro
      - ./grafana/dashboards:/var/lib/grafana/dashboards:ro
    depends_on:
      - prometheus

  prometheus:
    image: prom/prometheus:v3.13.4
    container_name: prometheus
    restart: unless-stopped
    command:
      - --config.file=/etc/prometheus/prometheus.yml
      - --storage.tsdb.path=/prometheus
      - --storage.tsdb.retention.time=30d
    ports:
      - "127.0.0.1:9090:9090"
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prometheus-data:/prometheus

  node-exporter:
    image: prom/node-exporter:v1.12.1
    container_name: node-exporter
    restart: unless-stopped
    pid: host
    command:
      - --path.rootfs=/host
    volumes:
      - /:/host:ro,rslave

secrets:
  grafana_admin:
    file: ./grafana_admin.txt

volumes:
  grafana-data:
  prometheus-data:
EOF

We don’t write the admin password into the compose file as plain text. It lives in a separate file with 600 permissions, and Grafana reads it via a variable with the __FILE suffix.


# On grafana-01: file with the admin password for the container
openssl rand -base64 24 | sudo tee /opt/monitoring/grafana_admin.txt > /dev/null
sudo chmod 600 /opt/monitoring/grafana_admin.txt
sudo cat /opt/monitoring/grafana_admin.txt

Prometheus config. Three targets: Prometheus itself, this host’s node_exporter and Grafana’s own metrics. The target names are the service names from the compose file, containers see each other by those names on Docker’s internal network.


# On grafana-01: Prometheus config
sudo tee /opt/monitoring/prometheus/prometheus.yml > /dev/null <<'EOF'
global:
  scrape_interval: 30s
  evaluation_interval: 30s

scrape_configs:
  - job_name: prometheus
    static_configs:
      - targets: ['prometheus:9090']

  - job_name: node
    static_configs:
      - targets: ['node-exporter:9100']
        labels:
          host: grafana-01

  - job_name: grafana
    static_configs:
      - targets: ['grafana:3000']
EOF

Now provisioning: the Prometheus data source Grafana will create on its own at startup. Note the address: http://prometheus:9090, not localhost. That’s exactly the mistake from cause number two.


# On grafana-01: automatically connect Prometheus to Grafana
sudo tee /opt/monitoring/grafana/provisioning/datasources/prometheus.yaml > /dev/null <<'EOF'
apiVersion: 1
datasources:
  - name: Prometheus
    uid: prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false
    jsonData:
      timeInterval: 30s
EOF

And the dashboard provider: Grafana will pick up JSON files from the directory and refresh them on change.


# On grafana-01: dashboard provider from files
sudo tee /opt/monitoring/grafana/provisioning/dashboards/files.yaml > /dev/null <<'EOF'
apiVersion: 1
providers:
  - name: files
    folder: Infrastructure
    type: file
    disableDeletion: true
    updateIntervalSeconds: 30
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards
EOF

Now let’s start it.


# On grafana-01: start the stack and verify
cd /opt/monitoring
sudo docker compose up -d
sudo docker compose ps
sudo docker compose logs grafana --tail 30
curl -s http://127.0.0.1:3000/api/health | jq
curl -s http://127.0.0.1:9090/api/v1/targets | jq '.data.activeTargets[] | {job: .labels.job, health}'

Result: three containers in running state, /api/health reports database ok, all three Prometheus targets in up state. Log into Grafana with admin and the password from grafana_admin.txt.

The grafana-data and prometheus-data volumes are named ones, Docker sets correct permissions on them automatically. If you mount a host directory instead, keep in mind: the Grafana process in the container runs as UID 472, and the directory needs to be writable by it. Otherwise the container crashes with a message that GF_PATHS_DATA isn’t writable.

Step 4. Connect Prometheus and server metrics

In the Docker setup, Prometheus is already connected through provisioning. In the package install, connect it manually or with the same YAML file in /etc/grafana/provisioning/datasources/.

In the UI: Connections → Data sources → Add data source → Prometheus. In the Prometheus server URL field, enter the address Prometheus is reachable at from the Grafana machine, for example http://10.10.0.30:9090. Click Save & test. You should see a success message for the request to the Prometheus API.

Install node_exporter on every monitored Linux server. Using the distro’s package is easiest, but the version there tends to be old. For consistency, grab the binary from GitHub and a systemd unit.


# On every monitored Linux server: node_exporter 1.12.1 as a service
cd /tmp
wget https://github.com/prometheus/node_exporter/releases/download/v1.12.1/node_exporter-1.12.1.linux-amd64.tar.gz
tar -xzf node_exporter-1.12.1.linux-amd64.tar.gz
sudo install -m 755 node_exporter-1.12.1.linux-amd64/node_exporter /usr/local/bin/
sudo useradd --system --no-create-home --shell /usr/sbin/nologin node_exporter
sudo tee /etc/systemd/system/node_exporter.service > /dev/null <<'EOF'
[Unit]
Description=Prometheus node_exporter
After=network-online.target

[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter
Restart=on-failure

[Install]
WantedBy=multi-user.target
EOF
sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
curl -s http://127.0.0.1:9100/metrics | head -n 5

Open port 9100 only for the Prometheus address. node_exporter returns a lot of system detail without authentication: kernel, filesystems, network interfaces.


# On each monitored server (UFW): only let Prometheus reach node_exporter
sudo ufw allow from 10.10.0.20 to any port 9100 proto tcp

A new server gets added to prometheus.yml with a line in targets. After editing, reload the config without restarting:


# On grafana-01 (Docker setup): reload the Prometheus config
sudo docker compose -f /opt/monitoring/docker-compose.yml kill -s SIGHUP prometheus

Step 5. Import a prebuilt Grafana dashboard

Why: building your first dashboard from scratch is slow and pointless. For node_exporter there’s a time-tested dashboard, Node Exporter Full, with ID 1860 in the Grafana dashboards catalog.

Import through the UI: Dashboards → New → Import, paste 1860 into the ID field, click Load, select the Prometheus source and click Import. If the Grafana server has no internet access, download the dashboard’s JSON on your workstation and upload it as a file through the same import window.

Through provisioning it’s done the same way with a file. Download the JSON and drop it into the dashboards directory. One catch: in the JSON from the catalog, the source is referenced by a variable that needs to be replaced with the source’s uid.


# On grafana-01: Node Exporter Full dashboard via provisioning
cd /tmp
curl -sL https://grafana.com/api/dashboards/1860/revisions/latest/download -o node-exporter-full.json
sed -i 's/${DS_PROMETHEUS}/prometheus/g' node-exporter-full.json
sudo cp node-exporter-full.json /opt/monitoring/grafana/dashboards/

In 30 seconds the dashboard shows up in the Infrastructure folder. In a package install, the directory path is set in its own provider file, for example /var/lib/grafana/dashboards.

Grafana 13 also has a built-in helper: the Suggested dashboards section matches ready-made dashboards to your data sources. For Prometheus it shows how well a dashboard matches your actual metrics. The feature is in public preview, but handy for getting started.

Step 6. Zabbix and Grafana: connect the Zabbix plugin

The Zabbix and Grafana combo is the most common reason admins end up in Grafana. Zabbix collects and alerts perfectly well, but its graphs and dashboards on a big screen look like a hello from 2012. The Zabbix plugin for Grafana pulls data through the Zabbix API and renders it properly.

On the Zabbix side. Create a dedicated read-only user. Don’t connect Grafana as Admin: the plugin stores the credentials, and any editor of the data source in Grafana gets full rights in Zabbix.

What to create in Zabbix Where Value
User group Users → User groups grafana-readers, “Read” permission on the needed host groups
User Users → Users grafana, User role, group grafana-readers
API token Users → API tokens For the grafana user, with an expiration date

Permissions on host groups matter here. Without them the API responds successfully but with an empty host list. In Grafana that looks like “everything connected, but there’s no data,” and it’s cause number five from the table.

On the Grafana side. Install the plugin. In the Docker setup it’s already installed via the GF_PLUGINS_PREINSTALL variable. On packages:


# On grafana-01 (package install): Zabbix plugin
sudo grafana cli --homepath /usr/share/grafana plugins install alexanderzobnin-zabbix-app
sudo systemctl restart grafana-server
sudo grafana cli --homepath /usr/share/grafana plugins ls

The plugin needs to be installed and also enabled. This is a separate action: Administration → Plugins and data → Plugins, find Zabbix and click Enable. Until the app is enabled, the Zabbix source type won’t appear in the list.

Enabling it can also be done via a provisioning file, so it survives container recreation:


# On grafana-01 (Docker setup): enable the Zabbix app at startup
sudo tee /opt/monitoring/grafana/provisioning/plugins/zabbix.yaml > /dev/null <<'EOF'
apiVersion: 1
apps:
  - type: alexanderzobnin-zabbix-app
    org_id: 1
    disabled: false
EOF

Now the data source. Through provisioning with an API token, no user password needed:


# On grafana-01 (Docker setup): Zabbix source with an API token
sudo tee /opt/monitoring/grafana/provisioning/datasources/zabbix.yaml > /dev/null <<'EOF'
apiVersion: 1
datasources:
  - name: Zabbix
    uid: zabbix
    type: alexanderzobnin-zabbix-datasource
    url: http://10.10.0.10/api_jsonrpc.php
    editable: false
    jsonData:
      authType: token
      trends: true
      trendsFrom: '7d'
      trendsRange: '4d'
      cacheTTL: '1h'
      timeout: 30
    secureJsonData:
      apiToken: $ZABBIX_API_TOKEN
EOF
sudo chmod 640 /opt/monitoring/grafana/provisioning/datasources/zabbix.yaml

The $ZABBIX_API_TOKEN construct is an environment variable substitution: Grafana expands such variables in provisioning files. The variable is already passed to the container in the compose file, all that’s left is to put the actual token in an .env file next to the compose file and lock it down with 600 permissions.


# On grafana-01 (Docker setup): Zabbix token in .env and restart Grafana
echo 'ZABBIX_API_TOKEN=paste_token_here_from_zabbix' | sudo tee /opt/monitoring/.env > /dev/null
sudo chmod 600 /opt/monitoring/.env
cd /opt/monitoring && sudo docker compose up -d grafana

Let’s go over the trends parameters separately, they have a big effect on speed. Zabbix keeps detailed history for a short while, and after that only hourly trends. With trends: true, the plugin uses trends instead of history for periods longer than 7 days. A dashboard covering a month opens in a second instead of a minute, without loading down the Zabbix database.

Through the UI it’s the same: Connections → Data sources → Add → Zabbix. API URL, Auth type: API token, paste the token, enable Trends, click Save & test. A successful test shows the Zabbix API version.

The Dashboards tab of the Zabbix source has the plugin’s ready-made dashboards. Import Zabbix Server Dashboard: it immediately shows whether the integration works. After that, the most useful thing in the plugin is the Problems panel. These are current Zabbix problems with filters by group and severity, convenient for showing on a screen for an on-call shift.

Good news for those who want to keep alerting in Zabbix: you don’t need to move it. Zabbix alerts, Grafana shows. Each tool does what it’s best at.

Step 7. Direct connection to the Zabbix database for heavy dashboards

When a dashboard has hundreds of metrics spanning a month, the Zabbix API starts to slow down. The plugin can pull history directly from the Zabbix database, while the API is only used for the list of hosts and items. This is enabled by the Direct DB Connection option in the Zabbix source settings.

For this you need a dedicated Zabbix database user with read-only access to the history and trends tables. Never connect Grafana to the Zabbix database under Zabbix’s own account: one wrong SQL query in a panel and you’re writing into the production database.

Read-only, only the tables you need
The user for Grafana should only have SELECT rights on the history and trends tables. No write permissions and no permissions on the Zabbix user and settings tables. Only allow the connection from the Grafana server’s address.

-- On the Zabbix DB server (PostgreSQL), as superuser: a read-only user
CREATE ROLE grafana_ro WITH LOGIN PASSWORD 'StrongPassword';
GRANT CONNECT ON DATABASE zabbix TO grafana_ro;
GRANT USAGE ON SCHEMA public TO grafana_ro;
GRANT SELECT ON history, history_uint, trends, trends_uint TO grafana_ro;

# pg_hba.conf on the Zabbix DB server: access only from the Grafana address
host    zabbix    grafana_ro    10.10.0.20/32    scram-sha-256

# On the Zabbix DB server: apply pg_hba.conf without restarting
sudo -u postgres psql -c "SELECT pg_reload_conf();"

In Grafana, add a PostgreSQL source with this user, and in the Zabbix source enable Direct DB Connection and select it. For MySQL the logic is the same: CREATE USER tied to the Grafana address and GRANT SELECT on the four tables.

Step 8. Your own Grafana dashboard with variables

Prebuilt dashboards are great for getting started, but sooner or later you need your own: one screen showing exactly what matters to your team. The main trick is variables. One dashboard serves all your servers through a dropdown, instead of twenty copies with different names.

Create a dashboard: Dashboards → New → New dashboard. Open the dashboard settings, Variables tab, add a variable:

Field Value Why
Name instance The name the variable is substituted under into queries
Type Query The list comes from the data source instead of being typed by hand
Data source Prometheus Where to pull values from
Query label_values(node_uname_info, instance) All servers that node_exporter is reporting from
Multi-value Enabled Allows picking several servers at once

Now the panels. PromQL queries you need on almost any dashboard for Linux servers:


# CPU load in percent
100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle", instance=~"$instance"}[5m])) * 100)

# Memory usage in percent
(1 - node_memory_MemAvailable_bytes{instance=~"$instance"} / node_memory_MemTotal_bytes{instance=~"$instance"}) * 100

# Filesystem usage in percent, excluding temporary filesystems
100 - (node_filesystem_avail_bytes{instance=~"$instance", fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes{instance=~"$instance", fstype!~"tmpfs|overlay"} * 100)

# Inbound traffic by interface, bits per second
rate(node_network_receive_bytes_total{instance=~"$instance", device!~"lo|veth.*|docker.*"}[5m]) * 8

Note the =~ instead of =. With Multi-value enabled, the variable turns into a regular expression like srv1|srv2, and exact equality won’t match against it. This is the classic reason a panel shows data for one server and nothing when you select two.

Set the measurement unit in the panel settings: Standard options → Unit, Percent (0-100) for percentages, bits/sec for traffic. Thresholds: green up to 70, yellow up to 90, red above. I set a 90% threshold for disk everywhere by default: better an extra yellow color than a call on Saturday.

For a Zabbix source, queries aren’t built with PromQL but by selecting Group, Host, Item in the editor. Variables work the same way: type Query, source Zabbix, a query like Group: Linux servers, Host: /.*/. The plugin understands regular expressions in slashes.

Export the saved dashboard to JSON: Share → Export → Save to file. Keep the export in Git. That gives you a change history, and the dashboard can be restored in a minute. Grafana 13 also has built-in Git Sync for this, now generally available in all editions.

Step 9. HTTPS through nginx

Why: Grafana over plain HTTP sends passwords and session cookies in cleartext. Plus nginx gives you a single port 443 facing outward and closes off 3000.


# On grafana-01: nginx
sudo apt-get install -y nginx

The config below assumes you already have a certificate, internal or from Let’s Encrypt. The /api/live/ block is needed for live updates over WebSocket: without it, dashboards still work, but the logs fill up with connection errors and live panels stop working.


# On grafana-01: nginx virtual host for Grafana
sudo tee /etc/nginx/sites-available/grafana.conf > /dev/null <<'EOF'
map $http_upgrade $connection_upgrade {
    default upgrade;
    ''      close;
}

server {
    listen 80;
    server_name grafana.example.local;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    http2 on;
    server_name grafana.example.local;

    ssl_certificate     /etc/ssl/certs/grafana.crt;
    ssl_certificate_key /etc/ssl/private/grafana.key;
    ssl_protocols TLSv1.2 TLSv1.3;

    location / {
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_pass http://127.0.0.1:3000;
    }

    location /api/live/ {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
        proxy_pass http://127.0.0.1:3000;
    }
}
EOF
sudo ln -s /etc/nginx/sites-available/grafana.conf /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx

After this, enable http_addr = 127.0.0.1 and cookie_secure = true in grafana.ini, check root_url and restart Grafana. The firewall only opens 80 and 443.


# On grafana-01: apply the settings and close off everything else
sudo systemctl restart grafana-server
sudo ufw allow 443/tcp
sudo ufw allow 80/tcp
sudo ufw allow from 10.10.0.0/24 to any port 22 proto tcp
sudo ufw enable
sudo ss -tlnp | grep -E ':3000|:443'

Result: Grafana listens on 3000 only on 127.0.0.1, nginx listens on 443 on all interfaces.

Step 10. Grafana on PostgreSQL instead of SQLite

Why: SQLite is fine for one instance and a hundred dashboards. Problems start when you run multiple Grafana instances behind a load balancer, have a lot of users, or save things frequently: you start getting database is locked errors. PostgreSQL removes these issues and backs up with the standard tools alongside your other databases.


# On the DB server (or on grafana-01): database and dedicated user for Grafana
sudo -u postgres psql -c "CREATE ROLE grafana WITH LOGIN PASSWORD 'StrongPassword';"
sudo -u postgres psql -c "CREATE DATABASE grafana OWNER grafana;"

# /etc/grafana/grafana.ini on grafana-01: database section
[database]
type = postgres
host = 127.0.0.1:5432
name = grafana
user = grafana
password = StrongPassword
ssl_mode = disable

A dedicated user with rights only on its own database is a must. Don’t give Grafana the postgres superuser. If the database is on a different machine, set ssl_mode = require and only allow access from the Grafana address in pg_hba.conf.

Here’s the key point: changing the database type doesn’t migrate data. A new Grafana on PostgreSQL starts out empty. Do the migration ahead of time: export dashboards through the API as shown in the backup section, then load them back in. Data sources are easier to just describe through provisioning than to migrate.

Step 11. Grafana API: service account and token

The API is needed for dashboard backups, automation and checks from scripts. Don’t access the API as the admin user: create a service account instead. Administration → Users and access → Service accounts → Add service account, role Viewer for reading or Editor for importing, then Add service account token.


# On any machine with access to Grafana: check the token and list dashboards
export GF_URL="https://grafana.example.local"
export GF_TOKEN="glsa_paste_token_here"
curl -s -H "Authorization: Bearer $GF_TOKEN" "$GF_URL/api/search?type=dash-db" | jq -r '.[] | "\(.uid)  \(.title)"'

Result: a list of uids and dashboard names. An empty array means the service account has no permissions on the folders.

Verification: how to make sure Grafana is working correctly

“The UI loads” isn’t a real check. Verify layer by layer: process, database, data sources, data, external access.


# On grafana-01: process, port and health
systemctl is-active grafana-server
sudo ss -tlnp | grep 3000
curl -s http://127.0.0.1:3000/api/health | jq

In the Docker setup, use docker compose ps instead of systemctl. A /api/health response with database ok means Grafana is alive and the database is reachable.

Data sources are checked through the API with the same request the Save & test button runs:


# On any machine with access to Grafana: health of all data sources
for uid in $(curl -s -H "Authorization: Bearer $GF_TOKEN" "$GF_URL/api/datasources" | jq -r '.[].uid'); do
  echo -n "$uid: "
  curl -s -H "Authorization: Bearer $GF_TOKEN" "$GF_URL/api/datasources/uid/$uid/health" | jq -r '.status + " " + .message'
done

The service account needs Editor or Admin role for this check, Viewer can’t see data source settings. You should get OK for every source. Any ERROR with text tells you exactly where to look.

Prometheus and its targets:


# On grafana-01: Prometheus target status
curl -s http://127.0.0.1:9090/api/v1/targets | jq -r '.data.activeTargets[] | "\(.labels.job) \(.labels.instance) \(.health) \(.lastError)"'

All targets should be up, the error field empty. A target in down with a connection refused error means node_exporter isn’t running or is blocked by a firewall.

Grafana logs. In the package install, the file is /var/log/grafana/grafana.log, in Docker it’s the container logs.


# On grafana-01: errors in the Grafana log
sudo grep -E 'level=(error|warn)' /var/log/grafana/grafana.log | tail -n 20
sudo docker compose -f /opt/monitoring/docker-compose.yml logs grafana --tail 50 | grep -E 'level=(error|warn)'

And external access: open https://grafana.example.local from your workstation, log in, open the imported dashboard and switch the range to Last 1 hour. The graphs are filled in, the instance variable shows the servers.

Grafana is working if
The service is running, /api/health returns database ok, all data sources pass the check, Prometheus targets are up, and a dashboard shows data for the last hour. The UI loads over HTTPS, and port 3000 isn’t reachable from outside.

Complications: Grafana errors and how to fix them

It’s not as bad as you think. It’s actually worse: in Grafana, a data source error and a query error look the same, an empty panel. So let’s start with the flowchart, then get into specifics.

%%{init: {
  'theme': 'base',
  'themeVariables': {
    'primaryColor': '#ffffff',
    'primaryTextColor': '#1e293b',
    'primaryBorderColor': '#94a3b8',
    'lineColor': '#64748b',
    'fontSize': '15px',
    'fontFamily': 'ui-sans-serif, system-ui, sans-serif'
  },
  'flowchart': {'curve': 'linear', 'nodeSpacing': 50, 'rankSpacing': 50}
}}%%
flowchart TD
    A["Panel shows No data"] --> B{"Does the source pass the test?"}
    B -->|no| C["Check the URL and access from the Grafana server"]
    B -->|yes| D{"Does the query work in Explore?"}
    D -->|no| E["Fix the query or labels"]
    D -->|yes| F{"Is the time range and time zone correct?"}
    F -->|no| G["Fix the range and time"]
    F -->|yes| H["Check the dashboard variables"]
    style A fill:#f8fafc,stroke:#f97316,stroke-width:2px,color:#9a3412
    style B fill:#f8fafc,stroke:#3b82f6,stroke-width:2px,color:#1e40af
    style D fill:#f8fafc,stroke:#3b82f6,stroke-width:2px,color:#1e40af
    style F fill:#f8fafc,stroke:#3b82f6,stroke-width:2px,color:#1e40af
    style C fill:#f8fafc,stroke:#22c55e,stroke-width:2px,color:#15803d
    style E fill:#f8fafc,stroke:#22c55e,stroke-width:2px,color:#15803d
    style G fill:#f8fafc,stroke:#22c55e,stroke-width:2px,color:#15803d
    style H fill:#f8fafc,stroke:#22c55e,stroke-width:2px,color:#15803d

The main diagnostic tool is the Explore section. It runs the same query as a panel, but without visualization settings and variables. If there’s data in Explore but not on the panel, the problem is with the panel or variables. If there’s no data there either, it’s in the source or the query.

Error Cause Fix
No data on a panel Query, time range, variables or source Follow the flowchart above, start with Explore
Bad Gateway or connection refused when testing the source localhost in the URL while Grafana runs in Docker, firewall Use a service name or host IP, test with curl from inside the container
No Zabbix source type available Plugin installed but not enabled Administration → Plugins → Zabbix → Enable
Zabbix: empty host list Zabbix user has no permissions on host groups “Read” permission for the grafana-readers group
grafana-cli doesn’t work on 13.x The separate binary is no longer supported Use grafana cli with the subcommand
451 geofence blocked when downloading Geo-restriction on Grafana Labs servers Your own mirror or manual package transfer
Endless redirect or blank page behind nginx root_url doesn’t match the real address Align root_url and server_name

Source in Docker: connection refused

Cause: the source URL has localhost or 127.0.0.1 in it. For Grafana in a container, that’s the container itself. Fix: if Prometheus is in the same compose file, use the service name. If the service runs on the host, use the host’s network IP or host.docker.internal with the appropriate setting.


# On grafana-01: check source reachability FROM INSIDE the Grafana container
sudo docker exec grafana wget -qO- http://prometheus:9090/-/healthy
sudo docker exec grafana wget -qO- http://10.10.0.10/api_jsonrpc.php --post-data='{"jsonrpc":"2.0","method":"apiinfo.version","params":{},"id":1}' --header='Content-Type: application/json-rpc'

If there’s no response from inside the container, Grafana won’t get one either. Fix the network, not Grafana.

Zabbix plugin: “Not authorized” or empty lists

Cause: the token expired, or the user has no rights on the API or on the host groups. With Zabbix 7.0, API access is controlled by the user’s role: if the API is disabled or restricted to a limited set of methods in that role, the plugin gets rejected.


# On grafana-01: check the Zabbix token and host visibility
export ZBX_TOKEN="paste_token_here"
curl -s -X POST http://10.10.0.10/api_jsonrpc.php \
  -H 'Content-Type: application/json-rpc' \
  -H "Authorization: Bearer $ZBX_TOKEN" \
  -d '{"jsonrpc":"2.0","method":"host.get","params":{"output":["host"],"limit":5},"id":1}' | jq

An authorization error points to a problem with the token or role. An empty result means no permissions on host groups. A list of hosts means Zabbix is fine, look for the problem in the source setup in Grafana.

Old dashboards stop opening after an upgrade

Cause: Grafana 12 completely removed support for AngularJS plugins. Old panels like Worldmap or outdated third-party graphs stop rendering. Built-in legacy panels like the old Grafana Graph get migrated automatically, third-party Angular plugins don’t.

Fix: before upgrading, go through your dashboards and replace Angular panels with modern equivalents. The built-in Grafana Advisor in the 13 branch helps find them: it checks for outdated plugins and sources with errors.


# On grafana-01: list installed plugins and their versions
sudo grafana cli --homepath /usr/share/grafana plugins ls

Forgot the admin password

Cause is simple enough. The fix is a reset from the server console, no data loss.


# On grafana-01 (package install): reset the admin password
sudo grafana cli --homepath /usr/share/grafana --config /etc/grafana/grafana.ini admin reset-admin-password 'NewStrongPassword'

# On grafana-01 (Docker setup): same thing inside the container
sudo docker exec -it grafana grafana cli admin reset-admin-password 'NewStrongPassword'

In the Docker setup, the GF_SECURITY_ADMIN_PASSWORD variable only works when the database is first created. You can’t change the password through it on a running Grafana, and that’s a frequent source of confusion.

Container won’t start: GF_PATHS_DATA is not writable

Cause: instead of a named volume, a host directory is mounted at /var/lib/grafana, for example /opt/monitoring/grafana-data, and it’s owned by root. Grafana in the container runs as UID 472 and can’t write there.


# On grafana-01: grant permissions on the Grafana data directory on the host
sudo chown -R 472:0 /opt/monitoring/grafana-data
sudo docker compose -f /opt/monitoring/docker-compose.yml up -d grafana

database is locked in the logs

Cause: SQLite can’t keep up with concurrent writes, usually during a bulk dashboard import, with a large number of users, or with the database on a slow network disk. If it’s a one-off error during an import, just retry. If it happens regularly, move to PostgreSQL as in step 10.


# On grafana-01: how often the database lock shows up
sudo grep -c 'database is locked' /var/log/grafana/grafana.log

Graphs are shifted in time

Cause: time isn’t synced on the source server or on Grafana, or the dashboard’s time zone is set to UTC while users expect local time. Check sync with timedatectl on all machines. The time zone is set in the dashboard settings, in the user’s profile, and globally in Administration → Default preferences.

Alternatives: Grafana equivalents and when you don’t need Grafana

Grafana isn’t the only way to look at metrics. Sometimes it’s installed just because it’s trendy, in places where built-in tools would be enough.

Tool When it fits How it compares to Grafana
Zabbix dashboards All monitoring is in Zabbix, a screen is needed for on-call staff Nothing extra to install, but visually poorer and less flexible
Prometheus web UI One-off queries and PromQL debugging Queries and simple graphs only, no dashboards
Kibana or OpenSearch Dashboards Primary data is logs in Elasticsearch or OpenSearch Stronger at log search, weaker at metrics
Yandex DataLens Business analytics, reports for managers A BI tool, not operational monitoring
Perses Dashboards as code in Kubernetes, a CNCF project Younger and leaner on plugins, but open and declarative
Netdata Detailed real-time metrics for a single server with zero configuration Fast to start, but not a unified dashboard for multiple sources
Grafana Cloud No desire to run your own infrastructure No administration needed, but monitoring data leaves your network

Why this guide still goes with self-hosted Grafana. It’s the only one on this list that works equally well with Prometheus, Zabbix, PostgreSQL, Loki and dozens of other sources. One screen for everything instead of five browser tabs, that’s the whole point of installing it.

Logs, traces and Alloy: when to expand the stack

A whole family of products has grown up around Grafana Labs. You don’t need to install all of them at once, but it’s worth knowing what each one does.

Product What it does When you need it
Loki Stores logs and feeds them into Grafana You need to view logs alongside metrics on the same dashboard
Tempo Stores distributed traces Microservices, OpenTelemetry, hunting for slow requests
Alloy Universal collector for metrics, logs and traces One agent instead of node_exporter, Promtail and an OpenTelemetry collector
Mimir Scalable metrics storage, Prometheus-compatible Millions of series and long retention

Grafana Alloy replaces Grafana Agent and Promtail, both of which are deprecated. If you’re starting to collect logs into Loki today, use Alloy, not the Promtail you see in old articles. At the time of publishing, Alloy 1.18 is current, check the latest release in the Alloy documentation.

For Proxmox there are two paths. The built-in Proxmox VE metrics server sends data to InfluxDB or Graphite, and Grafana draws it. Or use the Proxmox VE by HTTP template in Zabbix, and then Proxmox data reaches Grafana through the Zabbix plugin. If Zabbix is already in place, the second path is simpler: no need for another storage backend.

Prevention: monitoring, backup, security and upgrades for Grafana

A Grafana instance that nobody monitors falls over at the worst possible moment. Usually five minutes before a manager opens a dashboard in a meeting. Murphy’s law in monitoring works down to the second.

Monitoring Grafana itself

Grafana exposes its own metrics in Prometheus format at /metrics. In the Docker setup, Prometheus already scrapes them, the grafana target in prometheus.yml. In a package install, add a target with address 127.0.0.1:3000. Metrics worth alerting on:

Metric What it shows When to worry
up{job=”grafana”} Whether Grafana is responding Value of 0
grafana_http_request_duration_seconds Response time of the UI and API A several-fold jump in the median
grafana_datasource_request_total Requests to data sources by response code Growth in error responses
process_resident_memory_bytes Process memory Steady growth with no drops

If Zabbix is your primary monitoring tool, add an HTTP availability check for Grafana there too: a web scenario or an HTTP agent item against https://grafana.example.local/api/health with a trigger on a response code other than 200. The thing monitoring Grafana shouldn’t live inside Grafana.

Backup

What to back up Path How often Why
Grafana database /var/lib/grafana/grafana.db or the PostgreSQL database Daily Dashboards, users, sources, alerts
Configuration /etc/grafana On change and daily grafana.ini, provisioning
Dashboards as JSON Export via the API Daily Restoring a single dashboard without rolling back the whole database
Docker stack /opt/monitoring, excluding volumes On change Compose file, configs, provisioning
Don't copy the SQLite file on a running Grafana
A plain copy of grafana.db while Grafana is running can produce a corrupted copy if Grafana is writing to the database at that moment. Use SQLite’s built-in backup command: it makes a consistent snapshot without stopping the service.

# On grafana-01: script to back up the database, configs and dashboards
sudo tee /usr/local/sbin/grafana-backup.sh > /dev/null <<'EOF'
#!/bin/bash
set -euo pipefail
DIR=/var/backups/grafana/$(date +%F)
mkdir -p "$DIR/dashboards"
# Consistent SQLite snapshot
sqlite3 /var/lib/grafana/grafana.db ".backup '$DIR/grafana.db'"
# Configuration
tar -czf "$DIR/etc-grafana.tar.gz" /etc/grafana
# Dashboards as JSON via the API
source /etc/grafana-backup.env
for uid in $(curl -s -H "Authorization: Bearer $GF_TOKEN" "$GF_URL/api/search?type=dash-db" | jq -r '.[].uid'); do
  curl -s -H "Authorization: Bearer $GF_TOKEN" "$GF_URL/api/dashboards/uid/$uid" | jq '.dashboard' > "$DIR/dashboards/$uid.json"
done
find /var/backups/grafana -mindepth 1 -maxdepth 1 -type d -mtime +14 -exec rm -rf {} +
EOF
sudo chmod 750 /usr/local/sbin/grafana-backup.sh
printf 'GF_URL=http://127.0.0.1:3000\nGF_TOKEN=glsa_service_account_token\n' | sudo tee /etc/grafana-backup.env > /dev/null
sudo chmod 600 /etc/grafana-backup.env
sudo /usr/local/sbin/grafana-backup.sh && ls -lh /var/backups/grafana/$(date +%F)/

# On grafana-01: back up daily at 03:15
echo '15 3 * * * root /usr/local/sbin/grafana-backup.sh' | sudo tee /etc/cron.d/grafana-backup

For Grafana on PostgreSQL, replace the sqlite3 line with pg_dump -Fc for the grafana database. Pull the copy off the server from a central location, via rsync from a backup server. A backup on the same disk protects against a deleted dashboard, not against a dead disk.

Full restore: stop Grafana, put back grafana.db and /etc/grafana, set ownership to grafana, start it up. Restoring a single dashboard is simpler, just import the JSON file through the UI or call the API.


# On grafana-01: full restore from a backup
sudo systemctl stop grafana-server
sudo cp /var/backups/grafana/2026-10-20/grafana.db /var/lib/grafana/grafana.db
sudo tar -xzf /var/backups/grafana/2026-10-20/etc-grafana.tar.gz -C /
sudo chown grafana:grafana /var/lib/grafana/grafana.db
sudo systemctl start grafana-server

Autostart

Package install: the systemctl enable from step 3 already enabled autostart. Add a safety net for crashes through a drop-in, without touching the package’s unit:


# On grafana-01: auto-restart Grafana on crash
sudo mkdir -p /etc/systemd/system/grafana-server.service.d
printf '[Service]\nRestart=on-failure\nRestartSec=10s\n' | sudo tee /etc/systemd/system/grafana-server.service.d/restart.conf
sudo systemctl daemon-reload
systemctl show grafana-server -p Restart

Docker setup: restart: unless-stopped in the compose file and the docker service enabled. Check that Docker starts on boot: systemctl is-enabled docker.

Security

Grafana holds credentials for every connected data source. Compromising Grafana means handing over the keys to Prometheus, Zabbix and databases. A minimal security checklist:

  • Firewall: only 443 and 80 facing outward, SSH only from the admin network, port 3000 closed off.
  • Changed admin password, closed sign-up, disabled anonymous access.
  • Dedicated read-only users in Zabbix and in the backing databases.
  • HTTPS through nginx and cookie_secure.
  • fail2ban on SSH and on failed Grafana login attempts via the nginx log.
  • SSH keys only, no root login.
  • Daily off-server backups.

Grafana has built-in brute force protection: after a series of failed attempts, login is temporarily blocked for that user. Don’t disable it. fail2ban adds IP-level blocking at the firewall layer.


# On grafana-01: fail2ban for SSH
sudo apt-get install -y fail2ban
printf '[sshd]\nenabled = true\nbackend = systemd\nmaxretry = 5\nbantime = 1h\n' | sudo tee /etc/fail2ban/jail.d/sshd.local
sudo systemctl enable --now fail2ban
sudo fail2ban-client status sshd
Don't lock yourself out
Before disabling password login, verify key-based login from a second session and keep the first one open until you’ve confirmed it works.

# On grafana-01: SSH keys only
printf 'PermitRootLogin no\nPasswordAuthentication no\nKbdInteractiveAuthentication no\n' | sudo tee /etc/ssh/sshd_config.d/10-hardening.conf
sudo sshd -t && sudo systemctl reload ssh

Upgrading Grafana without surprises

Grafana ships often: a new minor branch roughly every two months, security patches even more frequently. You need to upgrade, but in an orderly way.

What to check before upgrading. Read the What’s new and breaking changes section for the target version in the Grafana upgrade guide. Run Grafana Advisor and confirm all plugins are compatible. Back up the database with the script above.

Database migrations are irreversible
At startup, a new Grafana version runs migrations on its database. An older version won’t start against an already-migrated database. So a rollback always means the old package plus a database backup taken before the upgrade. No backup, no rollback.

# On grafana-01 (packages): upgrade with a backup
sudo /usr/local/sbin/grafana-backup.sh
sudo apt-get update
apt-cache policy grafana
sudo apt-get install -y grafana
sudo systemctl status grafana-server --no-pager
curl -s http://127.0.0.1:3000/api/health | jq -r .version

# On grafana-01 (Docker): upgrade by changing the tag
sudo /usr/local/sbin/grafana-backup.sh
sudo sed -i 's#grafana/grafana:13.2.4#grafana/grafana:NEW_VERSION#' /opt/monitoring/docker-compose.yml
cd /opt/monitoring && sudo docker compose pull grafana && sudo docker compose up -d grafana

In the Docker setup, the backup script needs adapting: the database lives in the grafana-data volume. The simplest way is to stop the container and copy the volume through a temporary container, or run sqlite3 .backup inside the volume.

Rolling back a package install:


# On grafana-01: roll back to the previous version and restore the database
sudo systemctl stop grafana-server
sudo apt-get install -y --allow-downgrades grafana=VERSION_BEFORE_UPGRADE
sudo cp /var/backups/grafana/DATE_BEFORE_UPGRADE/grafana.db /var/lib/grafana/grafana.db
sudo chown grafana:grafana /var/lib/grafana/grafana.db
sudo systemctl start grafana-server
sudo apt-mark hold grafana

Remove apt-mark hold once the issue is resolved: sudo apt-mark unhold grafana. Otherwise six months from now you’ll find a Grafana stuck on a version with an already-patched vulnerability. Legacy code isn’t scared of refactoring, but it does love unpatched security holes.

FAQ: common Grafana questions

Why doesn’t Grafana show data after setup?

Most often the data source has the wrong address, especially localhost while Grafana runs in Docker, or an unsuitable time range is selected. Check with the Save & test button in the source settings, then run the query in Explore. If there’s data in Explore, the problem is in the panel or dashboard variables.

How do you check that Grafana is working correctly?

Query /api/health: a database ok response means the service and database are fine. Then check the health of each source via /api/datasources/uid/ID/health and open a dashboard for the last hour. For ongoing monitoring, scrape Grafana’s own metrics from /metrics into Prometheus or check /api/health from Zabbix.

What do you do if you forgot the admin password in Grafana?

Reset it from the server console with the grafana cli admin reset-admin-password command and a new password. Dashboards and settings are unaffected. In Grafana 13 use the grafana cli subcommand specifically: the separate grafana-cli binary doesn’t work in this branch.

What’s the difference between Grafana and Zabbix?

Zabbix is a full monitoring system: it collects data itself, stores history, evaluates triggers and sends notifications. Grafana only visualizes data from other systems, including Zabbix. A good combo: Zabbix collects and alerts, Grafana shows nice, flexible dashboards through the Zabbix plugin.

Prometheus and Grafana: what’s the difference and do you need both?

Prometheus collects and stores metrics, evaluates rules and sends alerts through Alertmanager. Grafana builds dashboards from those metrics. Prometheus has a simple web UI for queries, but it’s not enough for dashboards, so the two are almost always installed together.

How do you connect Zabbix to Grafana?

Install the alexanderzobnin-zabbix-app plugin, enable it in Administration → Plugins and add a Zabbix source with the api_jsonrpc.php address. Use an API token for a dedicated Zabbix user with read-only rights on the needed host groups. Enable trends so long time ranges open fast.

Is Grafana free?

Yes, Grafana OSS is free and distributed under the AGPLv3 license. The Enterprise edition can also be installed for free, with paid features unlocked by a license key. For internal company monitoring, the OSS license limits are usually more than enough.

How do you install Grafana in Docker?

Use the grafana/grafana image with a pinned version, mount a volume at /var/lib/grafana, and publish port 3000. A ready-made compose file with a Grafana, Prometheus and node_exporter stack is in Path B of this article. The grafana/grafana-oss image stopped being updated as of the 12.4 branch, don’t use it.

Outlook: what works now

We installed Grafana 13 and connected it to two worlds: Prometheus with node_exporter for Linux metrics, and Zabbix through the plugin with an API token and trends. Grafana sits behind nginx with HTTPS, port 3000 is closed off externally, and passwords and tokens aren’t sitting in plaintext in compose files. There are ready-made dashboards, your own dashboard with variables covering all servers, and a clear path to PostgreSQL once SQLite starts feeling cramped.

From here Grafana takes care of itself: systemd or Docker bring it back up if it crashes, a nightly backup of the database and dashboards lands in a separate directory, and upgrading comes down to a backup and one command. The next logical step is logs in Loki via Alloy, alongside the metrics. Then during an incident, one screen shows both what broke and why.

Didn't work?
If panels are empty or a source doesn’t pass the test, leave a comment and we’ll sort it out. Include the Grafana version, your install method, the source type, and the error text from Save and test or from the Grafana log.

Оставайтесь на связи

Рецепты от IT-боли. Без воды, без рекламы, без маркетинговой шелухи.

Подписаться на IT-Аптеку →
Поделитесь:

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top