Monitoring Fluentd log ingestion with Prometheus

Monitoring a managed Elasticsearch / Opensearch cluster is relatively easy nowadays. If you’re using, for example, Amazon Opensearch, there are a lot of metrics that will give you an insight about how the cluster is performing. You can easily track and monitor indexing rate, http request status, CPU usage, memory usage… What these metrics don’t tell you is what could be the cause of a spike in indexing…