Monitoring Kubernetes

Splunk field extraction

Define search-time field extraction rules for container logs

To have Splunk parse fields out of your container logs at search time, target each extraction at the right containers by matching on Collectord’s source value.

Every container log Collectord forwards carries a structured source that encodes the container ID, container name, image name, pod name, namespace, and stream:

text
1/kubernetes/logs/{kubernetes_container_id}/{kubernetes_container_name}/{kubernetes_image_name}/{kubernetes_pod_name}/{kubernetes_namespace}.{docker_stream}

Match that structure in props.conf: a wildcard (*) stands for one segment, and ... skips several. That way you can scope an extraction to a single image, a specific container name, or any combination.

For example, to apply an nginx ingress controller access-log extraction to every container running registry.k8s.io/ingress-nginx/controller, whatever its container ID, container name, pod, namespace or stream:

props.conf ini
1[source::/kubernetes/logs/*/*/registry.k8s.io/ingress-nginx/controller:*/*/*]
2EXTRACT-nginx-ingress-controller-http = ^(?P<remote_addr>[^ ]+)\s+\-\s+\[(?P<proxy_add_x_forwarded_for>[^\]]+)\]\s+\-\s+(?P<remote_user>[^ ]+)\s+\[(?P<time_local>[^\]]+)[^"\n]*"(?P<request>[^"]+)"\s+(?P<status>\d+)\s+(?P<body_bytes_sent>\d+)\s+"(?P<http_referer>[^"]+)"\s+"(?P<http_user_agent>[^"]+)"\s+(?P<request_length>\d+)\s+(?P<request_time>[^ ]+)\s+\[(?P<proxy_upstream_name>[^\]]+)]\s+(?P<upstream_addr>[^\s]+)\s+(?P<upstream_response_length>\d+)\s+(?P<upstream_response_time>[^\s]+)\s+(?P<upstream_status>\d+)$

You can also override the source or sourcetype per pod or namespace, and key your extractions off those instead: see Splunk Indexes.