Kubernetes is the de facto container management system on the market today, scheduling containerized workloads to nodes and ensuring that they are running. But you still need to collect metrics and create alerts. 

Unlike in traditional application architectures, metrics endpoints are not static in Kubernetes. If a node goes down, Kubernetes simply schedules the containers to other nodes, which means the monitoring system has to continue collecting metrics from the new instances. In other words, you need a cloud-native and Kubernetes-compatible monitoring system, like Prometheus. 

Prometheus is the leading instrumentation, collection, and storage tool for applications running in Kubernetes. It’s the best available tool to store and query data, plus you can implement Grafana to visualize the data with themed dashboards. Prometheus can create a bottleneck when you need to keep data for a long time and run queries. Luckily, long-term storage solutions such as Thanos make it possible to scale Prometheus to store more data, compact it, and query quickly. Thanos and Prometheus complement each other for production-grade monitoring systems, optimizing visibility and alerts in the cloud-native world. 

In this blog, we’ll create a Thanos-backed Prometheus monitoring system for applications running on Kubernetes. The secret gem will be running Thanos with auto-scaling, giving you a cloud-native and scalable monitoring system with long-term storage, running on Kubernetes. 

Note: In order to continue with the tutorial, make sure you have the following tools installed:                     

  • Minikube version: v1.13.0
  • Helm version: v3.3.2
  • kubectl version: v1.19.0

Prometheus and Thanos in Action

Let’s start by creating a Kubernetes cluster in Minikube. The following command will set up a new Kubernetes cluster with a metrics-server enabled. In the following steps, you will use this add-on to collect metrics and auto-scale.

$ minikube start --addons=metrics-server
😄 minikube v1.13.0 on Darwin 10.15.6
✨ Automatically selected the docker driver. Other choices: hyperkit, virtualbox
👍 Starting control plane node minikube in cluster minikube
🔥 Creating docker container (CPUs=2, Memory=9000MB) ...
🐳 Preparing Kubernetes v1.19.0 on Docker 19.03.8 ...
🔎 Verifying Kubernetes components...
🌟 Enabled addons: default-storageclass, metrics-server, storage-provisioner
🏄 Done! kubectl is now configured to use "minikube" by default

Now you can install the prometheus-operator to create and manage Prometheus instances. The first step is to add the Helm chart repository of the operator:

$ helm repo add bitnami https://charts.bitnami.com/bitnami
"bitnami" has been added to your repositories

Now, create a namespace to group all monitoring applications:

$ kubectl create namespace kube-monitoring
namespace/kube-monitoring created

Install the Helm chart with the following command, which ensures that the chart is installed in the kube-monitoring namespace and that thanos.create is enabled:

$ helm install prometheus-operator \
    --set prometheus.thanos.create=true \
    --namespace kube-monitoring \
    bitnami/prometheus-operator

...
NAME: prometheus-operator
LAST DEPLOYED: Fri Sep 18 14:39:17 2020
NAMESPACE: kube-monitoring
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **

Watch the Prometheus Operator Deployment status using the command:
...

The installation’s long output is essential if you want to go through the pods and manually check their statuses. 

The next step is to deploy Thanos. To do this, create a values.yaml file with the following content:

objstoreConfig: |-
  type: s3
  config:
    bucket: thanos
    endpoint: {{ include "thanos.minio.fullname" . }}.kube-monitoring.svc.cluster.local:9000
    access_key: minio
    secret_key: minio123
    insecure: true
querier:
  dnsDiscovery:
    sidecarsService: prometheus-operator-prometheus-thanos
    sidecarsNamespace: kube-monitoring
  autoscaling:
    enabled: true
    minReplicas: 1
    maxReplicas: 3
    targetCPU: 50
    targetMemory: 50
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
bucketweb:
  enabled: true
compactor:
  enabled: true
storegateway:
  enabled: true
ruler:
  enabled: true
  alertmanagers:
    - http://prometheus-operator-alertmanager.kube-monitoring.svc.cluster.local:9093
  config: |-
    groups:
      - name: "metamonitoring"
        rules:
          - alert: "PrometheusDown"
            expr: absent(up{prometheus="kube-monitoring/prometheus-operator"})
metrics:
  enabled: true
  serviceMonitor:
    enabled: true
minio:
  enabled: true
  accessKey:
    password: "minio"
  secretKey:
    password: "minio123"
  defaultBuckets: "thanos"

There are two essential sections in the values: querier.autoscaling and querier.resources. In the first, it‘s configured that new instances will be created when the target CPU or memory of 50% is reached. In the second, 100m CPU and 128Mi memory are requested for the Thanos Querier component. 

Now it’s time to deploy the chart with the following command:

$ helm install thanos \
    --values values.yaml \
    --namespace kube-monitoring \
    bitnami/thanos
NAME: thanos
LAST DEPLOYED: Fri Sep 18 15:13:16 2020
NAMESPACE: kube-monitoring
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
** Please be patient while the chart is being deployed **

Thanos chart was deployed enabling the following components:
- Thanos Querier
- Thanos Bucket Web
- Thanos Compactor
- Thanos Ruler
- Thanos Store Gateway
...

Wait for a couple of seconds to ensure that all pods are in Running state in the kube-monitoring namespace:

$ kubectl get pods -n kube-monitoring
NAME                                                      READY   STATUS    RESTARTS   AGE
alertmanager-prometheus-operator-alertmanager-0           2/2     Running   0          99s
prometheus-operator-kube-state-metrics-7b7955685b-ftr4b   1/1     Running   0          2m21s
prometheus-operator-node-exporter-sbnvs                   1/1     Running   0          2m21s
prometheus-operator-operator-5b7b998fd6-45rjx             1/1     Running   0          2m21s
prometheus-prometheus-operator-prometheus-0               4/4     Running   1          98s
thanos-bucketweb-7894f5455f-fqlkq                         1/1     Running   0          2m15s
thanos-compactor-7db44cb7f8-slgth                         1/1     Running   0          2m15s
thanos-minio-c8ccdcdc6-cnw5m                              1/1     Running   0          2m15s
thanos-querier-74544d675d-ckcj6                           1/1     Running   0          2m15s
thanos-ruler-0                                            1/1     Running   0          2m14s
thanos-storegateway-0                                     1/1     Running   2          2m14s

When all the pods are running, you can port-forward the Querier component to access its UI from the browser:

$ kubectl port-forward --namespace kube-monitoring svc/thanos-querier 9090
Forwarding from 127.0.0.1:9090 -> 10902
Forwarding from [::1]:9090 -> 10902

Go ahead and open the address 127.0.0.1:9090 in your browser to access the Thanos Querier UI:

Fig 1: Thanos Querier

You can run queries on the UI and check for metrics such as up{container=”querier”} as well as check the number of running Querier containers:

Fig 2: Metrics in Querier

As expected, there is only one Querier container. 

Now, let’s create some load on the container and check how it scales up. In another terminal, run the following command to send a query expression to the Thanos Querier API endpoint in an endless loop:

while true; do curl -s "http://127.0.0.1:9090/api/v1/query?query=up%7Bcontainer%3D%22querier%22%7D"; done

In yet another terminal, run the following command to watch the pods increase:

$ kubectl get pods --namespace kube-monitoring --selector app.kubernetes.io/component=querier -w
NAME                              READY   STATUS    RESTARTS   AGE
thanos-querier-74544d675d-pn4qx   1/1     Running   0          1h15m
thanos-querier-74544d675d-xrrlk   0/1     Pending   0          0s
thanos-querier-74544d675d-xrrlk   0/1     Pending   0          0s
thanos-querier-74544d675d-p5zcf   0/1     Pending   0          0s
thanos-querier-74544d675d-p5zcf   0/1     Pending   0          0s
thanos-querier-74544d675d-xrrlk   0/1     ContainerCreating   0          0s
thanos-querier-74544d675d-p5zcf   0/1     ContainerCreating   0          0s
thanos-querier-74544d675d-p5zcf   0/1     Running             0          4s
thanos-querier-74544d675d-xrrlk   0/1     Running             0          4s

Within a couple of seconds, you’ll see three Querier pods running in the cluster. Since you are querying inside a loop continuously, the load on the instances increases and forces Kubernetes to run a new instance. 

If you want to check how Kubernetes manages horizontal scaling, use the following command:

$ kubectl get HorizontalPodAutoscaler --namespace kube-monitoring
NAME             REFERENCE                   TARGETS            MINPODS   MAXPODS   REPLICAS   AGE
thanos-querier   Deployment/thanos-querier   11%/50%, 167%/50%   1         3         3          1h18m

There should be one Horizontal Pod Autoscaler with the name thanos-querier; this shows you the target limits and current scaling status.

You can also use the Querier UI to check how many instances of itself are running with the up{container=”querier} expression:

Figure 3: Thanos metrics after auto-scaling

By sending the continuous queries, you have increased the load on the Querier component. Kubernetes’ Pod Autoscalers collected the metrics and scaled up the instances to three, as configured. If you need more scalability, you can specify this in the values.yaml file above or update the HorizontalPodAutoscaler resource. Querier components are stateless and can be scaled arbitrarily with HorizontalPodAutoscaler. Similarly, storage capacity is scalable, but it relies on external object storage systems. However, first-class horizontal sharding of rules and data is not considered for the time being in Thanos architecture.

Summary

In this article, you were able to create an auto-scaling, Thanos-backed Prometheus monitoring system on Kubernetes, giving you the best of all worlds. Kubernetes is designed to run scalable, reliable, and highly available containerized applications, so its monitoring system is expected to be the same. Meanwhile, Prometheus enables you to collect metrics in a cloud-native way, and Thanos makes Prometheus reliable and durable over the long run. 

With the auto-scaling configuration, this system is highly available, meaning you can now create and monitor production-grade applications with a self-sufficient and cloud-native approach.

To learn on how you can use Epsagon to monitor, start a free trial.

Read More:

What you need to know about Kubernetes monitoring

A complete guide to monitoring EKS

How to setup Kubernetes in a hybrid environment