In a normal production environment monitoring your application’s output inside a running container should be very straightforward. Logs are usually collected from stdout and made visible in Kibana and your metrics are safely pushed to some time-based storage (such as Graphite) and then accessible via tools like Grafana.
So far goes the theory – and then there is real life. Remember this one dynamic machine you’re qa’ing on or a shadow instance that you quickly set up a week ago? And all over sudden you find yourself in the situation where you have to debug running code.
As always: First things first. The easiest way is to ssh directly to the host machine where your container lives. If you’re running your solution in some cloud as I do it’s usually quite convenient to obtain access to the instance in question (gcloud in this case).
1 | gcloud compute --project {yourproject} ssh --zone {thezone} {instancename} |
From there it’s easy to start outside-container investigations. First of all we need a container id to debug on (especially if there are multiple replicas running).
1 | docker ps | grep {imageName} |
With all container ids in question it’s easy to pull logs out of the running instance:
1 | docker logs {containerId} |
Also often applications crash because of a faulty environment setup. I do recommend dumping all env vars known to the container just to make sure that there’s no typo and nothing go lost in the configuration:
1 | docker exec -it {containerId} env |
If all that hasn’t revealed any information yet, it’s time to go right into the container and check on the actual system.
1 | docker exec -it {containerId} /bin/bash |
From there you can pretty much everything. Try not to kill it, otherwise it might be taken down / restarted by your container manager (e.g. kubernetes).