I haven’t used nginx as a Kubernetes IngressController since 2020. So, the information here may be outdated. However, recent issues reporting the same problem as mine are regularly opened, so I think this is still the case.
Context
Just so we can agree right away on what we’re talking about, I obviously don’t advise against using nginx in production. nginx is a robust software that has proven itself over the years and that I still use in certain professional and personal contexts.
However, when I started working with Kubernetes, I (like many) read the documentation.
And at the time, regarding the use of Ingress, the official documentation highlighted the implementation of the nginx IngressController available on the Kubernetes Github github.com/kubernetes/ingress-nginx.
Interestingly, there is another repository provided by the company nginx inc, which I have never used github.com/nginxinc/kubernetes-ingress. which may not be affected. However, when we talk about the “nginx IngressController” most people are talking about the repo on the Kubernetes Github (source: me, don’t worry)
So what is the problem?
Now that we are clear on what we are talking about, I can tell you a little story.
At the time, I worked for an industrial company that hosted “in the cloud” software sold as SaaS.
Each customer had their own instance, hosted on a Kubernetes hosted on Azure.
At first, we had few customers, everything was fine. But as the number of customers grew, we started to get feedback from the business and support teams that, from time to time, the application temporarily lost connection to the server.
As a good sysadmin and after checking the absence of alerts on our side, we first blamed the user, the network, and the application itself (in that order). #Trollface
Jokes aside, the reality is that we didn’t really know where to start with the problem because we had no logs, no incidents and the outages were very rare at first and above all seemingly random.
Until the day when…
Until one day, due to CVE issues to patch, we redeployed a very large number of applications in a fairly short time slot.
At that point, the number of complaints increased sharply and allowed us to validate that the problem was not random and was indeed on our side.
We guessed that there was a correlation between deploying applications in Kubernetes and dropping connections (websockets) on the client side.
We are not alone
Once we knew that, we found the culprit pretty quickly. A quick Google search turned up several issues on the IngressController repository describing the same symptoms.
Digging a little deeper, we also found a very comprehensive article that goes into much better detail than I could have done on the problem.
What’s the problem???
TL;DR as the IngressController is implemented by default, the nginx configuration is reloaded every time a change occurs on the Kubernetes Services backends.
For those who are not yet familiar with these concepts, every time a Pod (your containerized app) is created or destroyed, nginx is reloaded.
Theoretically, this is done hot because nginx supports hot configuration reloads. Except that, as the IngressController is implemented, open websockets must be cut off…
To limit the damage, each time the configuration is reloaded, an nginx process with the new configuration is spawned in parallel with the old one which is kept for a maximum of 10 seconds to give all current connections a chance to close cleanly.
Solutions?
In his post, Daniel reviews the workarounds available to you to limit the damage.
The first thing we can do and that works well is to add the option --enable-dynamic-configurationthat makes the configuration no longer reloaded each time the backends on the services change (very frequent events) but only when creating/deleting services (the deployment of a new app).
This drastically limits the number of reloads, but is obviously not a solution.
The second lever put forward by Daniel is to increase the timeout by 10s ( --worker-shutdown-timeout) and to put something very large. However, he warns against this solution because we quickly end up with a duplication of nginx processes with X versions of the conf, which can saturate the RAM of your ingressController or your node…
Last idea, to limit the disruption caused by a change, we had considered segmenting by having an ingressController per namespace. Thus, in case of redeployment, only the applications of the namespace currently being deployed would be affected by the cut (or rather “workarounds”):
Limit reloads (painful)
Significantly increase (24h?) the timeouts on active connections
Segment namespaces (1 Ingress Controller per namespace, we limit the impact of a redeployment to one client)
But then what do we do?
As there is no real solution to date, the simplest solution is to simply not use the Kubernetes nginx IngressController 😉 and to prefer an implementation that manages this kind of hot case.
Even though I haven’t worked with it for a while, we had good results with Traefik at the time.
And if you know of others who handle this kind of case well, don’t hesitate to let me know :).