Deploy Prometheus Operator with Helm3 and Private Registry

In this Blog we will go through how to Deploy Prometheus Operator into an air-gapped Kubernetes cluster using Private Docker registry.

You can use any private registry for deploying, I am using Harbor docker registry, as it supports some advanced features like Vulnerability scanning.

What is Prometheus

Prometheus is an open-source system monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community. To emphasize this, and to clarify the project’s governance structure, Prometheus joined the Cloud Native Computing Foundation in 2016 as the second hosted the project, after Kubernetes.

Prometheus has become one of the most used tool for monitoring and alerting in Kubernetes.

Prometheus works well for recording any purely numeric time series. It fits both machine-centric monitoring as well as monitoring of highly dynamic service-oriented architectures. In a world of microservices, its support for multi-dimensional data collection and querying is a particular strength.

Prometheus Operator

In late 2016, CoreOS introduced the Operator pattern and released the Prometheus Operator as a working example of the pattern. The Prometheus Operator automatically creates and manages Prometheus monitoring instances.

The mission of the Prometheus Operator is to make running Prometheus on top of Kubernetes as easy as possible while preserving configuration as well as making the configuration Kubernetes native. https://github.com/coreos/kube-prometheus#quickstart

Prerequisites

Before you begin, you will need access to platform and required CLIs

1. Kubernetes cluster (kubectl setup)
2. Helm 3 CLI installed, follow steps to install helm CLI
3. Internal or private Docker registry
4. Docker daemon on your Workstation
5. Git CLI

Let’s get our hands dirty!!

• First log in to your private docker repository, with your registry URL.

docker login your-registry.com -u admin -p your-password

You are logging in to repository to push docker images required for Prometheus operator deployment.

• Clone Kube-prometheus Github repository, this repository will help to get commands to download the images, and push them to our internal registry.

git clone https://github.com/coreos/kube-prometheus.git

Replace our-repository.com with your own registry name and replace project with your own project name in the next steps.

cd kube-prometheus

export REGISTRY=your-repository.com/project
docker run --rm -v $(pwd):$(pwd) --workdir $(pwd) quay.io/coreos/jsonnet-ci jsonnet -J vendor -S --tla-str repository=$REGISTRY sync-to-internal-registry.jsonnet | sh

You will get similar output like below, which you can run below output in your terminal to download docker images, tag them and push them to your internal registry. Do not copy-paste below output

From the above output, tag for kube-state-metrics image won’t work, there is a pending pull request to fix it, so for now, you can run

docker pull quay.io/coreos/kube-state-metrics:v1.9.5
docker tag quay.io/coreos/kube-state-metrics:v1.9.5 your-registry.com/monitoring/kube-state-metrics:v1.9.5
docker push your-registry.com/monitoring/kube-state-metrics:v1.9.5

docker pull quay.io/prometheus/alertmanager:v0.20.0
docker tag quay.io/prometheus/alertmanager:v0.20.0 your-registry.com/monitoring/alertmanager:v0.20.0
docker push your-registry.com/monitoring/alertmanager:v0.20.0
docker pull jimmidyson/configmap-reload:v0.3.0
docker tag jimmidyson/configmap-reload:v0.3.0 your-registry.com/monitoring/configmap-reload:v0.3.0
docker push your-registry.com/monitoring/configmap-reload:v0.3.0
docker pull grafana/grafana:6.6.0
docker tag grafana/grafana:6.6.0 your-registry.com/monitoring/grafana:6.6.0
docker push your-registry.com/monitoring/grafana:6.6.0
docker pull quay.io/coreos/kube-rbac-proxy:v0.4.1
docker tag quay.io/coreos/kube-rbac-proxy:v0.4.1 your-registry.com/monitoring/kube-rbac-proxy:v0.4.1
docker push your-registry.com/monitoring/kube-rbac-proxy:v0.4.1
docker pull quay.io/coreos/kube-state-metrics:1.9.5
docker tag quay.io/coreos/kube-state-metrics:1.9.5 your-registry.com/monitoring/kube-state-metrics:1.9.5
docker push your-registry.com/monitoring/kube-state-metrics:1.9.5
docker pull quay.io/prometheus/node-exporter:v0.18.1
docker tag quay.io/prometheus/node-exporter:v0.18.1 your-registry.com/monitoring/node-exporter:v0.18.1
docker push your-registry.com/monitoring/node-exporter:v0.18.1
docker pull quay.io/prometheus/prometheus:v2.15.2
docker tag quay.io/prometheus/prometheus:v2.15.2 your-registry.com/monitoring/prometheus:v2.15.2
docker push your-registry.com/monitoring/prometheus:v2.15.2
docker pull quay.io/coreos/k8s-prometheus-adapter-amd64:v0.5.0
docker tag quay.io/coreos/k8s-prometheus-adapter-amd64:v0.5.0 your-registry.com/monitoring/k8s-prometheus-adapter-amd64:v0.5.0
docker push your-registry.com/monitoring/k8s-prometheus-adapter-amd64:v0.5.0
docker pull quay.io/coreos/prometheus-config-reloader:v0.38.1
docker tag quay.io/coreos/prometheus-config-reloader:v0.38.1 your-registry.com/monitoring/prometheus-config-reloader:v0.38.1
docker push your-registry.com/monitoring/prometheus-config-reloader:v0.38.1
docker pull quay.io/coreos/prometheus-operator:v0.38.1
docker tag quay.io/coreos/prometheus-operator:v0.38.1 your-registry.com/monitoring/prometheus-operator:v0.38.1
docker push your-registry.com/monitoring/prometheus-operator:v0.38.1

In addition to the above images, you need to upload another image for the Admission webhook, Grafana Sidecar image, tlsProxy.

docker pull jettech/kube-webhook-certgen:v1.2.0
docker tag jettech/kube-webhook-certgen:v1.2.0 your-registry.com/monitoring/kube-webhook-certgen:v1.2.0
docker push your-registry.com/monitoring/kube-webhook-certgen:v1.2.0

docker pull kiwigrid/k8s-sidecar:0.1.99
docker tag kiwigrid/k8s-sidecar:0.1.99 your-registry.com/monitoring/k8s-sidecar:0.1.99
docker push your-registry.com/monitoring/k8s-sidecar:0.1.99

docker pull squareup/ghostunnel:v1.5.2
docker tag squareup/ghostunnel:v1.5.2 your-registry.com//monitoring/ghostunnel:v1.5.2
docker push your-registry.com//monitoring/ghostunnel:v1.5.2

Now you should have your Docker images ready in your Internal Repository, for example my Harbor project looks like this

• In this step, you will be creating Kubernetes secret to pull Docker images from your internal Registry. You can also refer to official docs

kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>

You can verify the secret created using below command

Kubectl get secrets

Output should look something like

kubectl get secrets
NAME                  TYPE                                  DATA   AGE
default-token-xwww2   kubernetes.io/service-account-token   3      24h
regcred               kubernetes.io/dockerconfigjson        1      2s

Now you have your registry secret created at k8s cluster level, let’s move to the next part of installing prometheus-operator. Let’s move forward to the next step.

• In this step will be using Helm CLI to prepare our charts.

Let’s clone the Github repository, which hosts all the stable charts.

git clone https://github.com/helm/charts.git

cd charts/stable/prometheus-operator

Directory content will look something like

CONTRIBUTING.md
Chart.yaml
OWNERS
README.md
ci
crds
hack
requirements.lock
requirements.yaml
templates
values.yaml

cat requirements.yaml

dependencies:

  - name: kube-state-metrics
    version: "2.8.*"
    repository: https://kubernetes-charts.storage.googleapis.com/
    condition: kubeStateMetrics.enabled

  - name: prometheus-node-exporter
    version: "1.9.*"
    repository: https://kubernetes-charts.storage.googleapis.com/
    condition: nodeExporter.enabled

  - name: grafana
    version: "5.0.*"
    repository: https://kubernetes-charts.storage.googleapis.com/
    condition: grafana.enabled

The modified file is shown below

cat requirements.yaml

dependencies:

  - name: kube-state-metrics
    version: "2.8.*"
    repository: file://../kube-state-metrics
    condition: kubeStateMetrics.enabled

  - name: prometheus-node-exporter
    version: "1.9.*"
    repository: file://../prometheus-node-exporter
    condition: nodeExporter.enabled

  - name: grafana
    version: "5.0.*"
    repository: file://../grafana
    condition: grafana.enabled

Check Helm CLI version before moving on to next step. Make sure it is 3.x version. If you haven’t installed CLI check the installation guide.

helm version

version.BuildInfo{Version:"v3.1.1", GitCommit:"afe70585407b420d0097d07b21c47dc511525ac8", GitTreeState:"clean", GoVersion:"go1.13.8"}

Now let’s update the dependency lock file using Helm CLI. Run Below command

helm dependency update

Once the dependency lock file is updated, you can see the updated file with the below-shown content.

cat requirements.lock

dependencies:
- name: kube-state-metrics
  repository: file://../kube-state-metrics
  version: 2.8.7
- name: prometheus-node-exporter
  repository: file://../prometheus-node-exporter
  version: 1.9.1
- name: grafana
  repository: file://../grafana
  version: 5.0.26
digest: sha256:f78d95110f396c5963038222d3a59ac146cf6c20697b2f0417f467442e8b9b35
generated: "2020-05-25T16:29:10.600184+08:00"

• In this step, you will be creating an additional file custom-values.yml for image location pointing to the internal registry for various components used to Deploy Prometheus Operator.
  
  Replace your-registry.com/monitoring with your internal registry URL
  
  Create the below file in the prometheus-operator chart directory.

touch custom-values.yaml

cat <<EOF > custom-values.yaml
global:
  imagePullSecrets:
  - name: regcred

prometheusOperator:
  image:
    repository: your-registry.com/monitoring/prometheus-operator
    tag: v0.38.1
  tlsProxy:
    image:
      repository: your-registry.com/monitoring/ghostunnel
      tag: v1.5.2
  prometheusConfigReloaderImage:
    repository: your-registry.com/monitoring/prometheus-config-reloader
    tag: v0.38.1
  configmapReloadImage:
    repository: your-registry.com/monitoring/configmap-reload
    tag: v0.3.0
  admissionWebhooks:
     patch:
       image:
         repository: your-registry.com/monitoring/kube-webhook-certgen
         tag: v1.2.0

alertmanager:
  alertmanagerSpec:
    image:
      repository: your-registry.com/monitoring/alertmanager
      tag: v0.20.0

prometheus:
  prometheusSpec:
    image:
      repository: your-registry.com/monitoring/prometheus
      tag: v2.17.2

kube-state-metrics:
  serviceAccount:
    imagePullSecrets:
    - name: regcred
  image:
    repository: your-registry.com/monitoring/kube-state-metrics
    tag: v1.9.5

grafana:
  image:
    pullSecrets:
    - regcred
    repository: your-registry.com/monitoring/grafana
    tag: 6.6.0
  sidecar:
    image: your-registry.com/monitoring/k8s-sidecar:0.1.99

prometheus-node-exporter:
  serviceAccount:
    imagePullSecrets:
    - name: regcred
  image:
    repository: your-registry.com/monitoring/node-exporter
    tag: v0.18.1
EOF

• This is the final step to Deploy Prometheus operator using helm CLI. You need to run the below command from prometheus-operator directory.

helm install prometheus-operator . -f custom-values.yaml

• Time to check the status of our Setup, run below commands to get the pod & deployment status.

kubectl get pods

NAME                                                      READY   STATUS    RESTARTS   AGE
alertmanager-prometheus-operator-alertmanager-0           2/2     Running   0          2m8s
prometheus-operator-grafana-895fcb474-tsg8v               2/2     Running   0          2m13s
prometheus-operator-kube-state-metrics-86666459b6-5s8jw   1/1     Running   0          2m13s
prometheus-operator-operator-6b556bd579-jpmjh             2/2     Running   0          2m13s
prometheus-operator-prometheus-node-exporter-llhgj        1/1     Running   0          2m13s
prometheus-operator-prometheus-node-exporter-mqk4k        1/1     Running   0          2m13s
prometheus-operator-prometheus-node-exporter-pxsjp        1/1     Running   0          2m13s
prometheus-prometheus-operator-prometheus-0               3/3     Running   1          118s

kubectl get deployments

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
prometheus-operator-grafana              1/1     1            1           18m
prometheus-operator-kube-state-metrics   1/1     1            1           18m
prometheus-operator-operator             1/1     1            1           18m

• Let’s see how you can access our Grafana Dashboard, you can make good use of Port Forwarding.

kubectl port-forward deployment/prometheus-operator-grafana 8080:3000

You can access your Grafana dashboard at http://127.0.0.1:8080

The default user/password is admin/prom-operator, which you can easily change by adding Grafana.adminPassword to custom-values.yaml file.

You can also expose Grafana using Loadbalancer type service or using ingress, refer to Grafana charts repository for more modifications.

For more modification and advance configuration you can refer to below links:

• Prometheus Operator Stable Chart Repository
• Kube-Prometheus with Minikube
• Getting started with Prometheus