Troubleshooting

For OpenShift use oc tool and namespace collectorforopenshift-syslog

Verify configuration

Get the list of the pods

$ kubectl get pods -n collectorforkubernetes-syslog
NAME                                           READY     STATUS    RESTARTS   AGE
collectorforkubernetes-syslog-addon-857fccb8b9-t9qgq   1/1       Running   1          1h
collectorforkubernetes-syslog-master-bwmwr             1/1       Running   0          1h
collectorforkubernetes-syslog-xbnaa                    1/1       Running   0          1h

Considering that we have 3 different deployment types, the DaemonSet we deploy on Masters (collectorforkubernetes-syslog-master), the DaemonSet we deploy on non-master nodes (collectorforkubernetes-syslog) and one Deployment addon (collectorforkubernetes-syslog-addon) verify one node from each deployment (in example below change the pod names to the pods that are running on your cluster).

$ kubectl exec -n collectorforkubernetes-syslog collectorforkubernetes-syslog-addon-857fccb8b9-t9qgq -- /collectord verify
$ kubectl exec -n collectorforkubernetes-syslog collectorforkubernetes-syslog-master-bwmwr -- /collectord verify
$ kubectl exec -n collectorforkubernetes-syslog collectorforkubernetes-syslog-xbnaa -- /collectord verify

OpenShift

For each command you will see an output similar to

Version = 5.2.176
Build date = 181012
Environment = kubernetes


  General:
  + conf: OK
  + db: OK
  + db-meta: OK
  + instanceID: OK
    instanceID = 2LEKCFD4KT4MUBIAQSUG7GRSAG
  + license load: OK
    trial
  + license expiration: OK
    license expires 2018-11-12 15:51:18.200772266 -0500 EST
  + license connection: OK

  Kubernetes configuration:
  + api: OK
  + pod cgroup: OK
    pods = 18
  + container cgroup: OK
    containers = 39
  + volumes root: OK
  + runtime: OK
    docker

  Docker configuration:
  + connect: OK
    containers = 43
  + path: OK
  + cgroup: OK
    containers = 40
  + files: OK

  CRI-O configuration:
  - ignored: OK
    kubernetes uses other container runtime

  File Inputs:
  x input(syslog): FAILED
    no matches
  + input(logs): OK
    path /rootfs/var/log/

Errors: 1

With the number of the errors at the end. In our example we show output from minikube, where we see some invalid configurations, like

• input(syslog) - minikube does not persist syslog output to disk, we will not be able to see these logs in applicatione

If you find some error in the configuration, after applying the change kubectl apply -f ./collectorforkubernetes-syslog.yaml you will need to recreate pods, for that you can just delete all of them in our namespace kubectl delete pods --all -n collectorforkubernetes. The workloads will recreate them.

Describe command

When you apply annotations through the namespace, workload, configurations and pods it could be hard to track which annotations are applied to the Pod or Container. You can use a describe command of collectord to get information which annotations are used for the specific Pod. You can use any collectord Pod to run this command on the cluster

kubectl exec -n collectorforkubernetes-syslog collectorforkubernetes-syslog-master-4gjmc -- /collectord describe --namespace default --pod postgres-pod --container postgres

Collect diagnostic information

If you need to open a support case you can collect diagnostic information, including performance, metrics and configuration (excluding splunk URL and Token).

1. Collect diagnostics information run following command

Choose pod from which you want to collect a diag information.

The following command takes several minutes.

kubectl exec -n collectorforkubernetes-syslog collectorforkubernetes-syslog-master-bwmwr -- /collectord diag --stream 1>diag.tar.gz

You can extract a tar archive to verify the information that we collect. We include information about performance, memory usage, basic telemetry metrics, information file with the information of the host Linux version and basic information about the license.

2. Collect logs

kubectl logs -n collectorforkubernetes-syslog --timestamps collectorforkubernetes-syslog-master-bwmwr  1>collectorforkubernetes.log 2>&1

3. Run verify

kubectl exec -n collectorforkubernetes-syslog collectorforkubernetes-syslog-master-bwmwr -- /collectord verify > verify.log

4. Prepare tar archive

tar -czvf collectorforkubernetes-$(date +%s).tar.gz verify.log collectorforkubernetes.log diag.tar.gz

Links

• Installation
  • Forwarding container logs, application logs, host logs and audit logs
  • Test our solution with the embedded 30 days evaluation license.
• Collector Configuration (Kubernetes)
  • Collector configuration reference for Kubernetes clusters.
• Collector Configuration (OpenShift)
  • Collector configuration reference for OpenShift clusters.
• Annotations
  • Changing type and format of messages forwarded from namespaces, workloads and pods.
  • Forwarding application logs.
  • Multi-line container logs.
  • Fields extraction for application and container logs (including timestamp extractions).
  • Hiding sensitive data, stripping terminal escape codes and colors.
• Audit Logs
  • Configure audit logs.
  • Forwarding audit logs.
• Troubleshooting
• FAQ and the common questions
• License agreement
• Pricing
• Contact