Observability at Scale

Recently we spotted an article in a blog from a specialist observability software company called Coralogix talking about the problems of observability at scale. We were thrilled to see that they have used our architecture as a reference point for how to scale an observability platform. What surprised us even more was how much our architecture has changed since we first wrote about it.

So what does our current setup look like?

Observability Stack

Simpler, more performant, easier to maintain, and easier to scale than it used to be.

In this article we will go through our new setup, explaining why we decided to replace/upgrade some of its components. I highly recommend to read our previous article!

For simplicity, we haven’t drawn the sidecars in the diagram, nor the replica instances for both Prometheus and Alertmanager:

• Prometheus instances run with a config reloader sidecar plus a Thanos sidecar
• Alertmanager instances run with a config reloader sidecar
• Grafana runs with a dashboard reloader sidecar

We have also omitted kube-state-metrics and node exporter daemonset, both of them are deployed as part of kube-prometheus-stack (more about it later).

Prometheus Operator

The most important change is the introduction of Prometheus Operator, deployed using the kube-prometheus-stack chart. As the number of time series generated by our services grew, in great part driven by the introduction of Istio, we needed to scale Prometheus horizontally. We wanted to give Istio its own Prometheus instances configured in HA. Up until then, they way to do so was:

• Run two completely independent deployments of Prometheus (we use Helm).
• Each deployment had its own (duplicated) scrape configuration, alerts, and recording rules.
• Each deployment had its own Alertmanager, configured as peers for HA. All Alertmanager configuration was duplicated as well.
• Every time we needed to implement a change in any of these configuration…