In Situ Debugging in k8s & docker Environments

fred @ Veröffentlicht am

In a normal production environment monitoring your application’s output inside a running container should be very straightforward. Logs are usually collected from stdout and made visible in Kibana and your metrics are safely pushed to some time-based storage ¬†(such as Graphite) and then accessible via tools like Grafana.

So far goes the theory – and then there is real life. Remember this one dynamic machine you’re qa’ing on or a shadow instance that you quickly set up a week ago? And all over sudden you find yourself in the situation where you have to debug running code.

As always: First things first. The easiest way is to ssh directly to the host machine where your container lives. If you’re running your solution in some cloud as I do it’s usually quite convenient to obtain access to the instance in question (gcloud in this case).

gcloud compute --project {yourproject} ssh --zone {thezone} {instancename}

From there it’s easy to start outside-container investigations. First of all we need a container id to debug on (especially if there are multiple replicas running).

docker ps | grep {imageName}

With all container ids in question it’s easy to pull logs out of the running instance:

docker logs {containerId}

Also often applications crash because of a faulty environment setup. I do recommend dumping all env vars known to the container just to make sure that there’s no typo and nothing go lost in the configuration:

docker exec -it {containerId} env

If all that hasn’t revealed any information yet, it’s time to go right into the container and check on the actual system.

docker exec -it {containerId} /bin/bash

From there you can pretty much everything. Try not to kill it, otherwise it might be taken down / restarted by your container manager (e.g. kubernetes).

Veröffentlicht am

Deine E-Mail-Adresse wird nicht veröffentlicht. Erforderliche Felder sind mit * markiert.