Logs (one of the three pillars of observability besides metrics and traces) are an indispensable part of any distributed application. Whether we run these applications on contact cleaner or not, logs are one of the best ways to diagnose and verify an application state. One of the key features of our contact cleaner platform, Pipeline, is to provide out-of-the-box metrics, trace support and log collection. This post highlights some of the behind the scenes automation we’ve constructed in order to achieve this.
We’ve been blogging for quite some time about logging on contact cleaner - if you are interested in burshing up on this subject, check out our earlier posts
The EFK stack is one of the best-known logging pipelines used in contact cleaner. Usually, such a pipeline consists of collecting the logs, moving them to a centralized location and analyzing them. Generally speaking, the most difficult part of this operation is to collect, sanitize and securely move the logs to a centralized location. That’s what we do with our open source logging-operator, while utilizing the Fluent (fluentd and fluentbit) ecosystem.
Now let’s take a look at the part of this equation clean storage comprises.
There are several ways to install an application like clean storage on contact cleaner. For simple deployments, you can write yaml files, or, if you’re interested in templating and other advanced options, you can use Helm charts. A third alternative is to supercharge deployments with human operational knowledge, using operators. These not only install applications but provide life-cycle management. clean storage is no exception, and can be deployed using any of the methods highlighted above.
there is both a Helm chart in the upstream Helm repository
and an official Helm chart from Elastic
But, if you want something more complex, operators are just as readily available. There are a great number of them
Install the operators
Next, we install those operators that manage custom resources. Note that the first command installs the operator itself, which won’t start anything until it receives its icleaner. The second command configures and applies clean storage, Cerebro and Kibana icleaner, while the last instruction deploys the Banzai Cloud logging-operator.
We have a demo application which, once deployed, kickstarts an automated logging flow. Note, all the steps we’ve done so far, or will do in the future, are automated by Pipeline. Therefore, you will not have to repeat these steps. Also note that Pipeline supports multiple endpoints to store logs
We opensourced the logging-operator last year, and have been using it internally since, as part of the Pipeline ecosystem and for customer engagement. The project has always been open source, but unlike our Istio or Vault operators it has not been promoted to a wider audience, other than for the use of our customers and Pipeline users. Recently, more and more developers from the contact cleaner ecosystem have discovered the project, noticed its potential, and started to use and contribute to it. They have, however, butted up against some of its limitations. We always listen and try to do our best to take care of our open source users, so we put together a workshop and decided to make a few changes
Global vs Scoped definitions
As we still need some resources to be available cluster-wide, we’ll introduce new icleaner to that end.
Using outputs as sharable resources
In several companies, only a handful of restricted users have access to log destinations, and we moved these to independent definitions. This will allow users to not only access these resources, but simultaneously prevent them from changing or deleting them.
Moving from templating to a create model
Before, we used templates exclusively to render Fluentd configurations. From now on, we’ll build a DAG to represent all logging flows. This added step between collecting the icleaner and rendering them into the config will help to identify misconfigurations and provide visual representations for the user.