Prometheus is an open-source and community-driven system monitoring and alerting. All metrics will be pulled, saved on a time-series database, and queried to display the metrics needed for monitoring. This article will show you how to set up a Prometheus Server on RHEL 8 (or RHEL 8-based distributions such as Alma Linux, Rocky Linux, and CentOS Stream).

Pre-requisites

For this article, we just need one VM with:

• Registered and attached a subscription with Red Hat Enterprise Linux 8 licensed.

• Internet access for downloading dependencies is needed.

• Open access to port 9090 (default port for Prometheus).

Steps

1. Configure SELinux

   For your information, SELinux does not have a specific policy but you can use it along with SELinux enabled by following this article: Installing Prometheus with SELinux and passing this section.

   The step below will edit the SELinux configuration file and set it to permissive. Moreover, for current boot session we need to set permissive.

     vim /etc/selinux/config
   

    # This file controls the state of SELinux on the system.
    # SELINUX= can take one of these three values:
    #     enforcing - SELinux security policy is enforced.
    #     permissive - SELinux prints warnings instead of enforcing.
    #     disabled - No SELinux policy is loaded.
    SELINUX=permissive
    # SELINUXTYPE= can take one of these three values:
    #     targeted - Targeted processes are protected,
    #     minimum - Modification of targeted policy. Only selected processes are protected.
    #     mls - Multi Level Security protection.
    SELINUXTYPE=targeted
   

    setenforce permissive
   

2. Download and extract Prometheus dependency

   You can download freely from Prometheus’ GitHub release page and choose specific release version and environment. On this article, we using Prometheus 2.37.8 for Linux with AMD64 architecture.

    curl -L -o prometheus-2.37.8.linux-amd64.tar.gz https://github.com/prometheus/prometheus/releases/download/v2.37.8/prometheus-2.37.8.linux-amd64.tar.gz
    tar -xvf prometheus-2.37.8.linux-amd64.tar.gz https://github.com/prometheus/prometheus/releases/download/v2.37.8/prometheus-2.37.8.linux-amd64.tar.gz
   

3. Setup Prometheus user and directory

   We need to create a user for the Prometheus SystemD service. Moreover, we need to set up the downloaded and extracted files related to Prometheus, such as prometheus.yml (the default Prometheus configuration file), prometheus (the main binary file for running the server), and promtool (the binary file for validating any configuration for Prometheus), with all file ownership granted to the Prometheus user.

    useradd --no-create-home --shell /sbin/nologin prometheus
    mkdir /etc/prometheus /var/lib/prometheus
    touch /etc/prometheus/web.yml
    mv prometheus-2.37.8.linux-amd64/prometheus.yml /etc/prometheus/
    mv prometheus-2.37.8.linux-amd64/prometheus /usr/local/bin/
    mv prometheus-2.37.8.linux-amd64/promtool /usr/local/bin/
    chown prometheus:prometheus -R /etc/prometheus/ /usr/local/bin/prometheus /usr/local/bin/promtool /var/lib/prometheus
   

4. Configure basic authentication for Prometheus

   You may need to secure your Prometheus by adding an user and access Prometheus as pre-configured user and password. On Prometheus, all user credential will saved as a web config file and all password should be saved as Bcrypt hashes password. You can hash your password with Bcrypt using Bcrypt-Generator.com site or follow this article: Basic auth.

    vim /etc/prometheus/web.yml
   

    basic_auth_users:
      admin: [Bcrypt-hashes-passowrd]
   

5. Configure Prometheus configuration

   This configuration will add credential configuration for scraping jobs from Prometheus server metrics.

    vim /etc/prometheus/prometheus.yml
   

    # my global config
    global:
      scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
      evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
      # scrape_timeout is set to the global default (10s).
   
    # Alertmanager configuration
    alerting:
      alertmanagers:
        - static_configs:
            - targets:
              # - alertmanager:9093
   
    # Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
    rule_files:
      # - "first_rules.yml"
      # - "second_rules.yml"
   
    # A scrape configuration containing exactly one endpoint to scrape:
    # Here it's Prometheus itself.
    scrape_configs:
      # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
      - job_name: "prometheus"
   
        # metrics_path defaults to '/metrics'
        # scheme defaults to 'http'.
   
        static_configs:
          - targets: ["localhost:9090"]
        basic_auth:
          username: "admin"
          password: [Unhashes-password]
   

6. Setup Prometheus service

   Lastly, we should create a Prometheus’ service to run background. The prometheus service will enabled and run even the VM restarted.

    vim /etc/systemd/system/prometheus.service
   

    [Unit]
    Description=Prometheus
    Wants=network-online.target
    After=network-online.target
   
    [Service]
    User=prometheus
    Group=prometheus
    Type=simple
    ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --web.config.file /etc/prometheus/web.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries
   
    [Install]
    WantedBy=multi-user.target
   

    systemctl daemon-reload
    systemctl enable --now prometheus
   

   Reference

   • Devopscube, “How To Install And Configure Prometheus Monitoring System On Linux,” DevopsCube, Oct. 21, 2020. https://devopscube.com/install-configure-prometheus-linux

   • J. Mutai, “How To Install Prometheus on RHEL 8 / CentOS 8,” ComputingForGeeks, Apr. 26, 2022. https://computingforgeeks.com/how-to-install-prometheus-on-rhel-8

   • Leon, “Installing Prometheus with SELinux,” Kubesimplify, May 16, 2022. https://blog.kubesimplify.com/installing-prometheus-with-selinux

   • Prometheus, “Securing Prometheus API and UI Endpoints Using Basic Auth,” Prometheus, https://prometheus.io/docs/guides/basic-auth

Revision:

28/06/2023 - 10:53 PM: Add step to configuring Prometheus.

Kubernetes is a container orchestration tool for managing containerized workloads and services. For creating a Kubernetes cluster, you should set up a container runtime and other configurations to initialize it. With MicroK8s, you can initialize a Kubernetes cluster in a simple way for learning Kubernetes.

Prerequisites

Two or more virtual machines for 1 master node and 1 or more worker node(s) with:

• Ubuntu 20.04 or newer

• 2 GB memory or more

• 20GB storage or more

• Internet access

Steps

1. Installing MicroK8s on the master node and worker node(s)

   On the master node and worker node(s)

    sudo snap install microk8s --classic
    sudo usermod -aG microk8s $USER
    sudo chown -fR $USER ~/.kube
    su - $USER
   

2. Joining worker node(s) to master node

   On master node

    microk8s add-node
    ---output omitted---
    Use the '--worker' flag to join a node as a worker not running the control
    plane, eg: microk8s join [master-node-ip]:25000/[some_value]/[some_value]--worker
    ---output omitted-
   

   On worker node(s)

    microk8s join [master-node-ip]:25000/[some_value]/[some_value] --worker
    ---output omitted---
    Currently, this worker node is configured with the following kubernetes API server endpoints:
    - [master-node-ip] and port 16443, this is the cluster node contacted during the join operation.
    ---output omitted---
   

3. Enable add-on

   On master node

    microk8s.enable dns dashboard metrics-server rbac
   

4. Setup kubectl

   On master node

    sudo apt install apt-transport-https --yes
    sudo curl -fsSLo /usr/share/keyrings/kubernetes-archive-keyring.gpg https://packages.cloud.google.com/apt/doc/apt-key.gpg
    echo "deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
    sudo apt update
    sudo apt install kubectl -y
    microk8s.config > ~/.kube/config
    echo 'source <(kubectl completion bash)' >>~/.bashrc
   

5. Labeling and tainting node(s)

    kubectl label [master-node-hostname] node-role.kubernetes.io/master=""
    kubectl label [worker-node-hostname1] node-role.kubernetes.io/worker=""
    kubectl taint [master-node-hostname1] node-role.kubernetes.io/master:NoSchedule
    kubectl label [worker-node-hostname(n)] node-role.kubernetes.io/worker=""
    kubectl taint [master-node-hostname(n)] node-role.kubernetes.io/master:NoSchedule
   

As a container orchestration, Kubernetes help anyone who needs to deploy a container service in a declarative way. Kubernetes will manage and maintain every resource to meet the state needed by the developer. Developer simply defined the desired state by JSON or YAML file and sends it to Kubernetes API. In this article, I will guide you to deploy a Kubernetes cluster with CRI-O as container runtime on RHEL-based OS, Rocky Linux 9.1.

Installation

Pre-requisites

We need 1 node for the control-plane node and at least 1 node or more nodes for the worker node with the specifications:

• Control-plane node:

  • 2 core or more processor

  • 2 GB or more memory

  • Rocky Linux 9

• Worker node:

  • 4 core or more processor

  • 8 GB or more memory

  • Rocky Linux 9

Steps

Master Node

1. Configure firewall on every node

   You need to set firewall rules on the node(s) to open several inbound ports or connections between the Kubernetes cluster node based on Kubernetes Documentation.

    firewall-cmd --permanent --new-zone=kubernetes-master
    firewall-cmd --permanent --zone=kubernetes-master --add-service=ssh
    firewall-cmd --permanent --zone=kubernetes-master --add-port={6443,2379,2380,10250,10259,10257}/tcp
    firewall-cmd --reload
    firewall-cmd --permanent --zone=kubernetes-master --change-interface=enp1s0
    firewall-cmd --set-default-zone=kubernetes-master
   

2. Setup CRI-O

   CRI-O repository will be added to YUM depending Kubernetes version want to be installed by specifying the VERSION environment variable.

    export VERSION=1.26
    export OS=CentOS_8_Stream
    curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo
    yum install cri-o -y
    systemctl enable --now crio
   

3. Setup kubeadm, kubelet, and kubectl

   Kubernetes’ repository will be added to YUM and install kubelet, kubectl and kubeadm following the version of Kubernetes want to installed. See Kubernetes Releases.

    cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
    [kubernetes]
    name=Kubernetes
    baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
    enabled=1
    gpgcheck=1
    gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
    exclude=kubelet kubeadm kubectl
    EOF
    # Set SELinux in permissive mode (effectively disabling it)
    sudo setenforce 0
    sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
    yum install kubelet-1.26.5 kubectl-1.26.5 kubeadm-1.26.5 --disableexcludes=kubernetes
   

4. Configure prerequisites

    cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
    overlay
    br_netfilter
    EOF
   
    modprobe overlay
    modprobe br_netfilter
   
    # sysctl params required by setup, params persist across reboots
    cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
    net.bridge.bridge-nf-call-iptables  = 1
    net.bridge.bridge-nf-call-ip6tables = 1
    net.ipv4.ip_forward                 = 1
    EOF
   
    # Apply sysctl params without reboot
    sysctl --system
   

5. Create kubeadm’s and InitConfiguration and ClusterConfiguration file for master node

    vim kubeadm-worker-conf.yaml
   

    apiVersion: kubeadm.k8s.io/v1beta3
    kind: InitConfiguration
    nodeRegistration:
     name: "master-node0"
     criSocket: "/var/run/crio/crio.sock"
    ---
    apiVersion: kubeadm.k8s.io/v1beta3
    kind: ClusterConfiguration
    networking:
     serviceSubnet: "10.10.0.0/16"
     podSubnet: "10.10.10.0/24"
     dnsDomain: "jesaya.lab"
    kubernetesVersion: "v1.26.5"
    controlPlaneEndpoint: "[control-plane-ip]:6443"
    clusterName: "[cluster-name]"
   

6. Initialize Kubeadm

    systemctl enable kubelet.service
    kubeadm init --config kubeadm-cluster-conf.yaml
   

7. Check the Kubernetes has been initilized

    mkdir -p $HOME/.kube
    cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
    chown $(id -u):$(id -g) $HOME/.kube/config
    kubectl get node
   

Worker node(s)

1. Configure firewall on every node

   You need to set firewall rules on the node(s) to open several inbound ports or connections between the Kubernetes cluster node based on Kubernetes Documentation.

    firewall-cmd --permanent --new-zone=kubernetes-worker
    firewall-cmd --permanent --zone=kubernetes-worker --add-service=ssh 
    firewall-cmd --permanent --zone=kubernetes-worker --add-port={10250,30000-32767}/tcp
    firewall-cmd --reload
    firewall-cmd --permanent --zone=kubernetes-worker --change-interface=enp1s0
    firewall-cmd --set-default-zone=kubernetes-worker
   

2. Setup CRI-O

   CRI-O repository will be added to YUM depending Kubernetes version want to be installed by specifying the VERSION environment variable.

    export VERSION=1.26
    export OS=CentOS_8_Stream
    curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo
    yum install cri-o -y
    systemctl enable --now crio
   

3. Setup kubeadm, kubelet, and kubectl

   Kubernetes’ repository will be added to YUM and install kubelet, kubectl and kubeadm following the version of Kubernetes want to installed. See Kubernetes Releases.

    cat <<EOF | sudo tee /etc/yum.repos.d/kubernetes.repo
    [kubernetes]
    name=Kubernetes
    baseurl=https://packages.cloud.google.com/yum/repos/kubernetes-el7-\$basearch
    enabled=1
    gpgcheck=1
    gpgkey=https://packages.cloud.google.com/yum/doc/rpm-package-key.gpg
    exclude=kubelet kubeadm kubectl
    EOF
    # Set SELinux in permissive mode (effectively disabling it)
    sudo setenforce 0
    sudo sed -i 's/^SELINUX=enforcing$/SELINUX=permissive/' /etc/selinux/config
    yum install kubelet-1.26.5 kubeadm-1.26.5 --disableexcludes=kubernetes
   

4. Configure prerequisites needed on node

    cat <<EOF | sudo tee /etc/modules-load.d/k8s.conf
    overlay
    br_netfilter
    EOF
   
    modprobe overlay
    modprobe br_netfilter
   
    # sysctl params required by setup, params persist across reboots
    cat <<EOF | sudo tee /etc/sysctl.d/k8s.conf
    net.bridge.bridge-nf-call-iptables  = 1
    net.bridge.bridge-nf-call-ip6tables = 1
    net.ipv4.ip_forward                 = 1
    EOF
   
    # Apply sysctl params without reboot
    sysctl --system
   

5. Create kubeadm’s JoinConfiguration file for worker node

    vim kubeadm-worker-conf.yaml
   

    apiVersion: kubeadm.k8s.io/v1beta3
    kind: JoinConfiguration
    nodeRegistration:
     name: "worker-node0"
     criSocket: "/var/run/crio/crio.sock"
    discovery: 
      bootstrapToken: 
        token: "[bootstrap-token]"
        apiServerEndpoint: "[control-plane-ip]:6443"
        caCertHashes: 
        - "[CA-cert-Hash]"
   

6. Joining worker node

    systemctl enable kubelet.service
    kubeadm join --config kubeadm-worker-conf.yaml