Grafana/Prometheus/Loki: Linux Install

Having a home lab is great but being able to monitor everything is even better. In this example I am setting up Grafana, Prometheus, Loki, Promtail and Node_Exporter on a monitoring server and Promtail, Node_Exporter on a client. My Grafana server is running OEL 8.5 which is essentially Red Hat 8.x.

Components

Minisrv1 – i5 CPU, 8GB Mem, 64GB flash – Grafana Server

Minisrv2 – Celeron CPU, 4GB Mem, 64GB flash – Using as kickstart server

Installing Grafana

Grafana is the easiest to install and get up running, compared to some of the other components:

# Install Grafana
$ sudo dnf -y install grafana
$ sudo systemctl enable --now grafana-server.service 
 Synchronizing state of grafana-server.service with SysV service script with /usr/lib/systemd/systemd-sysv-install.
 Executing: /usr/lib/systemd/systemd-sysv-install enable grafana-server
 Created symlink /etc/systemd/system/multi-user.target.wants/grafana-server.service → /usr/lib/systemd/system/grafana-server.service

The default port used is 3000. If you have another process using this port, you’ll need to set custom port in Grafana configuration file /etc/grafana/grafana.ini.

http_port = 3000

# Your grafana-server service should show running state.
$ systemctl status grafana-server.service 
● grafana-server.service - Grafana instance
   Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2022-03-03 10:30:44 EST; 58min ago
     Docs: http://docs.grafana.org
 Main PID: 1705 (grafana-server)
    Tasks: 20 (limit: 48277)
   Memory: 152.7M
   CGroup: /system.slice/grafana-server.service
           ├─1705 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/var/run/grafana/grafana-server.pid --packaging=rpm cfg:default.paths.logs=/>           └─2252 /var/lib/grafana/plugins/performancecopilot-pcp-app/datasources/redis/pcp_redis_datasource_linux_amd6411510)
    Memory: 17.3M
    CGroup: /system.slice/grafana-server.service
            └─2974 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/var/run/grafana/grafana-Started Grafana instance.

Open firewall port for Grafana

If you have a running firewalld service, allow port 3000 for access to the dashboard from the network:

$ sudo firewall-cmd --add-port=3000/tcp --permanent
$ sudo firewall-cmd --reload

Access Grafana Dashboard

Grafana web dashboard is accessible on http://[Server IP|Hostname]:3000

Install Prometheus

$ sudo yum install prometheus -y

# Add the clients that you want information 
$ vi /etc/prometheus/prometheus.yml
# Global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. 
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute. 
  scrape_timeout: 15s  # scrape_timeout is set to the global default (10s).
# A scrape configuration containing exactly one endpoint to scrape:# Here it's Prometheus itself.
scrape_configs:
# The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
# metrics_path defaults to '/metrics'
# scheme defaults to 'http'.
    static_configs:
    - targets: ['localhost:9090']
  - job_name: 'node_exporter'
    static_configs:
    - targets: ['localhost:9100']
# Linux Servers
  - job_name: minisrv1-grafana
    static_configs:
      - targets: ['localhost:9100']
        labels:
          alias: grafana
  - job_name: minisrv2-kickstart
    static_configs:
      - targets: ['10.0.1.141:9100']
        labels:
          alias: kickstart

# Start and enable Prometheus

$ sudo systemctl start prometheus

$ sudo systemctl enable prometheus
$ sudo systemctl status prometheus
● prometheus.service - Prometheus Time Series Collection and Processing Server
   Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2022-03-10 07:57:29 EST; 2h 38min ago
 Main PID: 54216 (prometheus)
    Tasks: 10 (limit: 48277)
   Memory: 180.1M
   CGroup: /system.slice/prometheus.service
           └─54216 /usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path>

Add Prometheus data source

Go to your Grafana browser
Click on the Sprocket icon and Data Sources

3. Stick with the defaults, then click Save/Test

4. if Save/Test fails your install from the step above may have failed. check the status of the prometheus service.

$ systemctl status prometheus
● prometheus.service - Prometheus Time Series Collection and Processing Server
   Loaded: loaded (/etc/systemd/system/prometheus.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2022-03-10 07:57:29 EST; 2h 38min ago
 Main PID: 54216 (prometheus)
    Tasks: 10 (limit: 48277)
   Memory: 180.1M
   CGroup: /system.slice/prometheus.service
           └─54216 /usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path>

If it failed check your config file: /etc/prometheus/prometheus.yml.  Chances are there is something incorrect.  You can comment things out until you figure it out unless it's obvious.

Install Loki

# Download & Install Loki
$ sudo curl -O -L "https://github.com/grafana/loki/releases/download/v2.4.2/loki-linux-amd64.zip"
$ sudo unzip "loki-linux-amd64.zip"
$ sudo chmod a+x "loki-linux-amd64"
$ sudo ./loki-linux-amd64 -config.file=loki-local-config.yaml
$ sudo cp loki-linux-amd64 /usr/local/bin
ls -l /usr/local/bin/loki-linux-amd64

# Create config file
$ sudo mkdir /etc/loki
$ sudo vi /etc/loki/loki-local-config.yaml
server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://localhost:3100/loki/api/v1/push

scrape_configs:
- job_name: system
  static_configs:
  - targets:
      - localhost
    labels:
      job: varlogs
      __path__: /var/log/*log

# Create a startup service 
$ sudo vi /etc/systemd/system/loki.service

[Unit]
Description=Loki
Wants=network-online.target
After=network-online.target

[Service]
User=root
Group=root
Type=simple
ExecStart=/usr/local/bin/loki-linux-amd64 -config.file=/etc/loki/loki-local-config.yaml

[Install]
WantedBy=multi-user.target


# Enable and Start Loki

$ sudo systemctl enable loki


$ sudo systemctl start loki
$ sudo systemctl status loki
● loki.service - Loki
   Loaded: loaded (/etc/systemd/system/loki.service; enabled; vendor preset: disabled)
   Active: active (running) since Wed 2022-03-09 15:14:45 EST; 20h ago
 Main PID: 12889 (loki-linux-amd6)
    Tasks: 10 (limit: 48277)
   Memory: 51.8M
   CGroup: /system.slice/loki.service
           └─12889 /usr/local/bin/loki-linux-amd64 -config.file=/etc/loki/loki-local-config.yaml

Add Loki Datasource

As before with Prometheus we will install Loki as a datasource with defaults. Save and Test if it fails check to see if instance is running from above steps.