Outcold Solutions LLC

Monitoring Docker - Version 5

Monitoring Multiple Clusters and ACL

Identifying the clusters

Identify the cluster with the configuration

When you start collectorfordocker container specify the cluster name with the configuration

--env "COLLECTOR__CLUSTER=general__fields.docker_cluster=-"

For example

--env "COLLECTOR__CLUSTER=general__fields.docker_cluster=development"

(Obsolete) Defining labels for clusters

Most of our dashboards allow you to filter data based on the docker labels.

If you have two clusters prod and dev, you can add labels to the Docker daemon to identify different nodes. For example, if you configure Docker daemon with /etc/docker/daemon.json (Debian/Ubuntu), you can add a label to each node, similarly to

{
  "labels" : {
    "cluster" : "prod",
  }
}

In case if you configure Docker daemon with /etc/sysconfig/docker (common in CentOS/RHEL case with Docker 1.13), you can add

--label=cluster=prod

Restart the daemon

$ systemctl restart docker

Verify that Docker picked up the change

$ docker info | grep -A 1 Labels
Labels:
 cluster=prod

After that, you should be able to see labels in the application dashboards and filter with them.

ACL for Clusters

All searches in the application are powered by the macros. If you want to separate access to the data for specific clusters or containers you can define different target indexes for clusters or containers and update the macros to use these indexes.

For example, let's assume you have Admins, Team1 and Team2 organizations in your company. You want to make Admins see data from Production and Development environments, Team1 only data from Containers built by their Team, and Team2 only data from the Containers built by Team2.

You can define several indices

  • docker_prod_team1
  • docker_prod_team2
  • docker_prod
  • docker_dev_team1
  • docker_dev_team2
  • docker_dev

Create two HTTP Tokens. One for the Production cluster with the default index docker_prod, allow this Token to write to docker_prod_team1, docker_prod_team2. Another token for Development cluster with the default index docker_dev, allow this Token to write to docker_dev_team1, docker_dev_team2.

For Docker hosts running in Production environment use the First token, for hosts running Development environment use the Second token. Use annotations to override Indexes for containers built by Team1 and Team2 to redirect their data to indexes docker_prod_team1, docker_prod_team2, docker_dev_team1, docker_dev_team2.

In Splunk change the macros to always search in the indices index=docker_*. Create 3 roles in Splunk, one Admins, that have access to all created indices, second role Team1 with access to docker_prod_team1 and docker_dev_team1, and third role Team2 with access to docker_prod_team2 and docker_dev_team2. Now, depending who is logged in with Splunk you will see a different set of data in the application. Team1 and Team2 will not be able to see system-related information, only logs and metrics from their containers. Admins will be able to see all the information.

About Outcold Solutions

Outcold Solutions provides solutions for monitoring Kubernetes, OpenShift and Docker clusters in Splunk Enterprise and Splunk Cloud. We offer certified Splunk applications, which give you insights across all containers environments. We are helping businesses reduce complexity related to logging and monitoring by providing easy-to-use and deploy solutions for Linux and Windows containers. We deliver applications, which help developers monitor their applications and operators to keep their clusters healthy. With the power of Splunk Enterprise and Splunk Cloud, we offer one solution to help you keep all the metrics and logs in one place, allowing you to quickly address complex questions on container performance.