When running a global platform, insight is absolutely key to diagnosing and resolving problems. At Polystream we use a number of tools to help provide developers with these insights with Instana making a critical part of that toolkit.
One of my colleagues recently mentioned that he had noticed that our internal dashboard had started to feel slower after a recent release, although not a critical issue any sudden change, especially after a release, is worth investigation.
The first step in investigating this issue was confirming that there was in fact a problem and it wasn’t just user perception. Very quickly I was able to generate a graph in Instana which clearly showed that it was a real problem and that our latency was well outside what we would deem to be acceptable.
From here Instana makes it very easy to drill down and see each of the individual requests which shows that it’s not every request that is slow.
Then we can dig into an individual call and get a trace to work out exactly where the time is going.
This makes it very clear, we have an issue with data getting over the network. The Polystream platform runs on Kubernetes and uses Istio for communication between pods so my first thought was have we introduced something that is using a large amount of CPU causing the pods to get bottle necked? Fortunately Instana makes it a couple of clicks to get from here to the actual infrastructure running the pod that serviced that request and that’s where I found the problem.
Although the CPU usage of the pod was extremely low it was being continuously throttled by the resource limits we had placed on the docker container.
In the end, this came down to a simple miss-configuration of the CPU limits we were placing on our docker containers, however, identifying that as the cause would have been much more difficult if it wasn’t for Instana clearly showing us the problem and easily allowing us to jump between individual requests and the infrastructure serving that request.
Insight to everything happening on the platform is fundamental to Polystream being able to offer a global platform and Instana not only gives us that insight, but it also makes it easily accessible to everyone on the team.