Deploy Prometheus Operator with Helm3 and Private Registry
Table of Contents
In this Blog we will go through how to Deploy Prometheus Operator into an air-gapped Kubernetes cluster using Private Docker registry.
You can use any private registry for deploying, I am using Harbor docker registry, as it supports some advanced features like Vulnerability scanning.
What is Prometheus
Prometheus is an open-source system monitoring and alerting toolkit originally built at SoundCloud. Since its inception in 2012, many companies and organizations have adopted Prometheus, and the project has a very active developer and user community. To emphasize this, and to clarify the project’s governance structure, Prometheus joined the Cloud Native Computing Foundation in 2016 as the second hosted the project, after Kubernetes.
Prometheus has become one of the most used tool for monitoring and alerting in Kubernetes.
Prometheus works well for recording any purely numeric time series. It fits both machine-centric monitoring as well as monitoring of highly dynamic service-oriented architectures. In a world of microservices, its support for multi-dimensional data collection and querying is a particular strength.
Prometheus Operator
In late 2016, CoreOS introduced the Operator pattern and released the Prometheus Operator as a working example of the pattern. The Prometheus Operator automatically creates and manages Prometheus monitoring instances.
The mission of the Prometheus Operator is to make running Prometheus on top of Kubernetes as easy as possible while preserving configuration as well as making the configuration Kubernetes native. https://github.com/coreos/kube-prometheus#quickstart
Prerequisites
Before you begin, you will need access to platform and required CLIs
- Kubernetes cluster (kubectl setup)
- Helm 3 CLI installed, follow steps to install helm CLI
- Internal or private Docker registry
- Docker daemon on your Workstation
- Git CLI
Let’s get our hands dirty!!
- First log in to your private docker repository, with your registry URL.
docker login your-registry.com -u admin -p your-password
You are logging in to repository to push docker images required for Prometheus operator deployment.
- Clone Kube-prometheus Github repository, this repository will help to get commands to download the images, and push them to our internal registry.
git clone https://github.com/coreos/kube-prometheus.git
Replace our-repository.com with your own registry name and replace project with your own project name in the
next steps.
cd kube-prometheus
export REGISTRY=your-repository.com/project
docker run --rm -v $(pwd):$(pwd) --workdir $(pwd) quay.io/coreos/jsonnet-ci jsonnet -J vendor -S --tla-str repository=$REGISTRY sync-to-internal-registry.jsonnet | sh
You will get similar output like below, which you can run below output in your terminal to download docker images, tag them and push them to your internal registry. Do not copy-paste below output
From the above output, tag for kube-state-metrics image won’t work, there is a pending pull request to fix it, so for now, you can run
docker pull quay.io/coreos/kube-state-metrics:v1.9.5
docker tag quay.io/coreos/kube-state-metrics:v1.9.5 your-registry.com/monitoring/kube-state-metrics:v1.9.5
docker push your-registry.com/monitoring/kube-state-metrics:v1.9.5
docker pull quay.io/prometheus/alertmanager:v0.20.0
docker tag quay.io/prometheus/alertmanager:v0.20.0 your-registry.com/monitoring/alertmanager:v0.20.0
docker push your-registry.com/monitoring/alertmanager:v0.20.0
docker pull jimmidyson/configmap-reload:v0.3.0
docker tag jimmidyson/configmap-reload:v0.3.0 your-registry.com/monitoring/configmap-reload:v0.3.0
docker push your-registry.com/monitoring/configmap-reload:v0.3.0
docker pull grafana/grafana:6.6.0
docker tag grafana/grafana:6.6.0 your-registry.com/monitoring/grafana:6.6.0
docker push your-registry.com/monitoring/grafana:6.6.0
docker pull quay.io/coreos/kube-rbac-proxy:v0.4.1
docker tag quay.io/coreos/kube-rbac-proxy:v0.4.1 your-registry.com/monitoring/kube-rbac-proxy:v0.4.1
docker push your-registry.com/monitoring/kube-rbac-proxy:v0.4.1
docker pull quay.io/coreos/kube-state-metrics:1.9.5
docker tag quay.io/coreos/kube-state-metrics:1.9.5 your-registry.com/monitoring/kube-state-metrics:1.9.5
docker push your-registry.com/monitoring/kube-state-metrics:1.9.5
docker pull quay.io/prometheus/node-exporter:v0.18.1
docker tag quay.io/prometheus/node-exporter:v0.18.1 your-registry.com/monitoring/node-exporter:v0.18.1
docker push your-registry.com/monitoring/node-exporter:v0.18.1
docker pull quay.io/prometheus/prometheus:v2.15.2
docker tag quay.io/prometheus/prometheus:v2.15.2 your-registry.com/monitoring/prometheus:v2.15.2
docker push your-registry.com/monitoring/prometheus:v2.15.2
docker pull quay.io/coreos/k8s-prometheus-adapter-amd64:v0.5.0
docker tag quay.io/coreos/k8s-prometheus-adapter-amd64:v0.5.0 your-registry.com/monitoring/k8s-prometheus-adapter-amd64:v0.5.0
docker push your-registry.com/monitoring/k8s-prometheus-adapter-amd64:v0.5.0
docker pull quay.io/coreos/prometheus-config-reloader:v0.38.1
docker tag quay.io/coreos/prometheus-config-reloader:v0.38.1 your-registry.com/monitoring/prometheus-config-reloader:v0.38.1
docker push your-registry.com/monitoring/prometheus-config-reloader:v0.38.1
docker pull quay.io/coreos/prometheus-operator:v0.38.1
docker tag quay.io/coreos/prometheus-operator:v0.38.1 your-registry.com/monitoring/prometheus-operator:v0.38.1
docker push your-registry.com/monitoring/prometheus-operator:v0.38.1
In addition to the above images, you need to upload another image for the Admission webhook, Grafana Sidecar image, tlsProxy.
docker pull jettech/kube-webhook-certgen:v1.2.0
docker tag jettech/kube-webhook-certgen:v1.2.0 your-registry.com/monitoring/kube-webhook-certgen:v1.2.0
docker push your-registry.com/monitoring/kube-webhook-certgen:v1.2.0
docker pull kiwigrid/k8s-sidecar:0.1.99
docker tag kiwigrid/k8s-sidecar:0.1.99 your-registry.com/monitoring/k8s-sidecar:0.1.99
docker push your-registry.com/monitoring/k8s-sidecar:0.1.99
docker pull squareup/ghostunnel:v1.5.2
docker tag squareup/ghostunnel:v1.5.2 your-registry.com//monitoring/ghostunnel:v1.5.2
docker push your-registry.com//monitoring/ghostunnel:v1.5.2
Now you should have your Docker images ready in your Internal Repository, for example my Harbor project looks like this

- In this step, you will be creating Kubernetes secret to pull Docker images from your internal Registry. You can also refer to official docs
kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword> --docker-email=<your-email>
You can verify the secret created using below command
Kubectl get secrets
Output should look something like
kubectl get secrets
NAME TYPE DATA AGE
default-token-xwww2 kubernetes.io/service-account-token 3 24h
regcred kubernetes.io/dockerconfigjson 1 2s
Now you have your registry secret created at k8s cluster level, let’s move to the next part of installing prometheus-operator. Let’s move forward to the next step.
- In this step will be using Helm CLI to prepare our charts.
Let’s clone the Github repository, which hosts all the stable charts.
git clone https://github.com/helm/charts.git
cd charts/stable/prometheus-operator
Directory content will look something like
CONTRIBUTING.md
Chart.yaml
OWNERS
README.md
ci
crds
hack
requirements.lock
requirements.yaml
templates
values.yaml
requirements.yaml
defines the dependency of this particular chart on other Helm charts for additional functionalities.
As we are using offline mode for charts, we need to modify below file to point to local chart directory.
cat requirements.yaml
dependencies:
- name: kube-state-metrics
version: "2.8.*"
repository: https://kubernetes-charts.storage.googleapis.com/
condition: kubeStateMetrics.enabled
- name: prometheus-node-exporter
version: "1.9.*"
repository: https://kubernetes-charts.storage.googleapis.com/
condition: nodeExporter.enabled
- name: grafana
version: "5.0.*"
repository: https://kubernetes-charts.storage.googleapis.com/
condition: grafana.enabled
The modified file is shown below
cat requirements.yaml
dependencies:
- name: kube-state-metrics
version: "2.8.*"
repository: file://../kube-state-metrics
condition: kubeStateMetrics.enabled
- name: prometheus-node-exporter
version: "1.9.*"
repository: file://../prometheus-node-exporter
condition: nodeExporter.enabled
- name: grafana
version: "5.0.*"
repository: file://../grafana
condition: grafana.enabled
Check Helm CLI version before moving on to next step. Make sure it is 3.x version. If you haven’t installed CLI check the installation guide.
helm version
version.BuildInfo{Version:"v3.1.1", GitCommit:"afe70585407b420d0097d07b21c47dc511525ac8", GitTreeState:"clean", GoVersion:"go1.13.8"}
Now let’s update the dependency lock file using Helm CLI. Run Below command
helm dependency update
Once the dependency lock file is updated, you can see the updated file with the below-shown content.
cat requirements.lock
dependencies:
- name: kube-state-metrics
repository: file://../kube-state-metrics
version: 2.8.7
- name: prometheus-node-exporter
repository: file://../prometheus-node-exporter
version: 1.9.1
- name: grafana
repository: file://../grafana
version: 5.0.26
digest: sha256:f78d95110f396c5963038222d3a59ac146cf6c20697b2f0417f467442e8b9b35
generated: "2020-05-25T16:29:10.600184+08:00"
- In this step, you will be creating an additional file custom-values.yml for image location pointing to the internal registry for various components used to Deploy Prometheus Operator.
Replace your-registry.com/monitoring with your internal registry URL
Create the below file in the prometheus-operator chart directory.
touch custom-values.yaml
cat <<EOF > custom-values.yaml
global:
imagePullSecrets:
- name: regcred
prometheusOperator:
image:
repository: your-registry.com/monitoring/prometheus-operator
tag: v0.38.1
tlsProxy:
image:
repository: your-registry.com/monitoring/ghostunnel
tag: v1.5.2
prometheusConfigReloaderImage:
repository: your-registry.com/monitoring/prometheus-config-reloader
tag: v0.38.1
configmapReloadImage:
repository: your-registry.com/monitoring/configmap-reload
tag: v0.3.0
admissionWebhooks:
patch:
image:
repository: your-registry.com/monitoring/kube-webhook-certgen
tag: v1.2.0
alertmanager:
alertmanagerSpec:
image:
repository: your-registry.com/monitoring/alertmanager
tag: v0.20.0
prometheus:
prometheusSpec:
image:
repository: your-registry.com/monitoring/prometheus
tag: v2.17.2
kube-state-metrics:
serviceAccount:
imagePullSecrets:
- name: regcred
image:
repository: your-registry.com/monitoring/kube-state-metrics
tag: v1.9.5
grafana:
image:
pullSecrets:
- regcred
repository: your-registry.com/monitoring/grafana
tag: 6.6.0
sidecar:
image: your-registry.com/monitoring/k8s-sidecar:0.1.99
prometheus-node-exporter:
serviceAccount:
imagePullSecrets:
- name: regcred
image:
repository: your-registry.com/monitoring/node-exporter
tag: v0.18.1
EOF
- This is the final step to Deploy Prometheus operator using helm CLI. You need to run the below command from prometheus-operator directory.
helm install prometheus-operator . -f custom-values.yaml
- Time to check the status of our Setup, run below commands to get the pod & deployment status.
kubectl get pods
NAME READY STATUS RESTARTS AGE
alertmanager-prometheus-operator-alertmanager-0 2/2 Running 0 2m8s
prometheus-operator-grafana-895fcb474-tsg8v 2/2 Running 0 2m13s
prometheus-operator-kube-state-metrics-86666459b6-5s8jw 1/1 Running 0 2m13s
prometheus-operator-operator-6b556bd579-jpmjh 2/2 Running 0 2m13s
prometheus-operator-prometheus-node-exporter-llhgj 1/1 Running 0 2m13s
prometheus-operator-prometheus-node-exporter-mqk4k 1/1 Running 0 2m13s
prometheus-operator-prometheus-node-exporter-pxsjp 1/1 Running 0 2m13s
prometheus-prometheus-operator-prometheus-0 3/3 Running 1 118s
kubectl get deployments
NAME READY UP-TO-DATE AVAILABLE AGE
prometheus-operator-grafana 1/1 1 1 18m
prometheus-operator-kube-state-metrics 1/1 1 1 18m
prometheus-operator-operator 1/1 1 1 18m
- Let’s see how you can access our Grafana Dashboard, you can make good use of Port Forwarding.
kubectl port-forward deployment/prometheus-operator-grafana 8080:3000
You can access your Grafana dashboard at http://127.0.0.1:8080
The default user/password is admin/prom-operator, which you can easily change by adding Grafana.adminPassword to custom-values.yaml file.

You can also expose Grafana using Loadbalancer type service or using ingress, refer to Grafana charts repository for more modifications.
For more modification and advance configuration you can refer to below links:
- Prometheus Operator Stable Chart Repository
- Kube-Prometheus with Minikube
- Getting started with Prometheus