Getting logs and metrics in your Splunk cluster is just a first step to manage your logs. Next step is to build your own custom dashboard to be able to explore the data you forward to Splunk.

The collector has a special format for sources that it forwards from container logs. We on purpose decided to replace it from unused and non-interesting /var/lib/docker/containers/{container_id}/{container_id}-logs.json to something that we can leverage with Splunk. So instead we are sending logs with the format that includes container name, image name and more, depends on used orchestration. You can find the format definition in Docker, Kubernetes and OpenShift documentation.

With the following example, we show how you can leverage this knowledge to extract fields from your container logs and start building custom dashboards. We use Monitoring OpenShift as an example, but you can apply this to any of our applications.

At first, you need to find the logs and define the source rule. Using wildcards you can cover multiple sources, like all containers that created from a specific image like in the example below

In our deployment we have 6 pods with similar sources.

/openshift/5d9ab541136dfd1a6a41efe25b481a432147e7452b4dd0755c0cb666e925cb79/nodejs-ex/docker-registry.default.svc:5000/nodejs/nodejs-ex:latest/nodejs-ex-3-bgqvk/nodejs.stdout
/openshift/9c078f0896fbad08df0a1b10c2c26d40a3166f940f4d6c3f731469f1a9152e11/nodejs-ex/docker-registry.default.svc:5000/nodejs/nodejs-ex:latest/nodejs-ex-3-dfbxg/nodejs.stdout
/openshift/05cebcacac1ccc6cb768169f3fb5544f606eeca1615ab339e5e01156aa6fa56a/nodejs-ex/docker-registry.default.svc:5000/nodejs/nodejs-ex:latest/nodejs-ex-3-gbqmf/nodejs.stdout
/openshift/185fa1c7e42658666fe704bc5b64366544c17d4d4edb83356c0ee2ebb1f2df6b/nodejs-ex/docker-registry.default.svc:5000/nodejs/nodejs-ex:latest/nodejs-ex-3-c79zr/nodejs.stdout
/openshift/7e12a1738ac448537c724b34b9451b1541ec60a5abf82f5de9912166c7497f76/nodejs-ex/docker-registry.default.svc:5000/nodejs/nodejs-ex:latest/nodejs-ex-3-hlz88/nodejs.stdout
/openshift/b14c3eb86db75e4f006cdc86bb9cbd08450b327ac9955f0299288e94f0e9053d/nodejs-ex/docker-registry.default.svc:5000/nodejs/nodejs-ex:latest/nodejs-ex-3-drvj7/nodejs.stdout

Based on the format for OpenShift container logs we can define the source by applying wildcards to container id and pod suffix.

/openshift/*/nodejs-ex/docker-registry.default.svc:5000/nodejs/nodejs-ex:latest/nodejs-ex*/nodejs.stdout

Using this pattern we can define field extraction in Splunk. For that go to the Settings, Fields, Field extractions and choose Open Field Extractor. Change data type to source and paste the source defined above. Verify that you can see all the logs that you expect and define fields extraction by following the wizard.

With this simple approach you can easily extract fields from container logs and start building custom dashboards.