Log Aggregation With EFK

Logs are underrated in many enterprise environments. Logs are often completely ignored, and only noticed when disk space is low. At that point, they’re usually deleted without review.

Sometimes logs are seen as a way to troubleshoot operational problems. Logs can be a good source of forensic information for determining what happened after an incident. However, we think that proactive logging enables improving business decisions. Logs, and in particular application logs, can contain a wide range of information that is not available otherwise.

Why are logs ignored? Log analysis isn’t easy. Effective log analysis take some work. Logs come in a variety of shapes and forms, and it can be difficult to extract information from them. The volume of logs generated by distributed applications can be overwhelming and difficult to correlate.

Wouldn’t it be great if you could aggregate the logs from multiple locations(servers, containers or even deleted pods) in a single place? Imagine how useful it would be if you could index them and run fast queries to get the data you’re looking for?

At giffgaff, we’ve decided to use EFK Stack ( Elasticsearch, Fluentd, Kibana) to provide such capability. EFK allows you to collect, index, search, and visualize log data. The EFK stack is a modified version of the ELK stack and is comprised of: