Understanding Kubernetes OOM & CPU Throttling and Optimizing Application Resources

When working with Kubernetes, Out of Memory (OOM) errors and CPU throttling are major headaches in resource handling in cloud applications. Why so?

CPU and Memory requirements in cloud applications are more important than ever because they are directly related to your cloud usage costs.

With limits and requirements, you can configure your pods to allocate memory and CPU resources to prevent resource exhaustion and regulate cloud usage costs.

• In case a Node does not have enough resources, Pods can be dropped through preemption or node pressure.
• When a process runs out of memory (OOM), it is killed because it doesn’t have the necessary resources.
• In case the CPU consumption is higher than the actual limit, the process will start to be throttled.

However, how can you proactively monitor your Kubernetes Pods closure with OOM and CPU throttling?

Kubernetes OOM Every container in a Pod needs memory to run. The Kubernetes limit is set per container by either the Pod definition or the Deployment definition. All modern Unix systems have a way to kill processes in case they need to reclaim memory. This will be marked as Error 137 or OOMKilled .

This Code 137 means that the process has used more memory than allowed and must terminate.

This is a feature included in Linux, where the kernel sets the oom_score value for the process running in the system. Additionally, it allows setting a value called oom_score_adj, which is used by Kubernetes to enable Quality of Service. It also has an OOM Killer feature, which will review processes and terminate those that are using more memory than necessary.

Note that in Kubernetes, a process can reach any of the following limits:

• Kubernetes limit set on container.n
• The Kubernetes ResourceQuota is located on the namespace.
• The actual memory size of Node.
• Kubernetes OOM chart

Over-commit memory

The limit can be higher than required, so the sum of all the limits can be higher than the capacity of the node. This is called overcommit and it is very common. In fact, if all containers use more memory than required, it can exhaust the memory in node. This usually causes the death of some Pods to free some memory.

Monitoring Kubernetes OOM

When using the node exporter in Prometheus, there is a metric called node_vmstat_oom_kill. It’s important to keep track of when an OOM kill occurs, but you may want to go ahead and have a vision of such an event before it happens.

Instead, you can check how close a process is to the Kubernetes limit:

Kubernetes CPU Throttling

CPU throttling is a behavior in which processes are slowed down when they are about to reach some resource limit. Similar to the memory case, these limits could be:

Kubernetes limits set on containers

The Kubernetes ResourceQuota is located on the namespace. Actual memory size of nodes.

Think of the following analogy. We have a highway with several means of transport:

• CPU is the way.
• Vehicles represent the process, where each has a different size.
• Multiple lanes represent having multiple cores. The requirement would be a dedicated road, such as a bike lane.
• Throttling here is expressed in terms of traffic congestion: eventually, all processes will run, but everything will be slower.

CPU Process in Kubernetes

CPU is handled in shared Kubernetes. Each CPU core is divided into 1024 shares, which are then divided among all running processes using the cgroups (control group) feature of the Linux kernel.

If the CPU can handle all current processes then no action is needed. If processes are using more than 100% CPU, then sharing will be done. Like any Linux Kernel, Kubernetes uses CFS (Completely Logical Scheduler) mechanism, so processes with more shares get more CPU time.

Unlike memory, Kubernetes will not kill Pods because of CPU throttling.

You can check CPU stats in /sys/fs/cgroup/cpu/cpu.stat

CPU overcommit

As we have seen in the article about limits and requests, it is important to set a limit or a request when we want to limit the resource consumption of our processes. However, be careful when setting the total number of requests to be larger than the actual CPU size, as this means that every container must have a guaranteed amount of CPU.

Kubernetes CPU Throttling Monitoring

You can check how close a process is to the Kubernetes limit:

In case we want to keep track of the number of adjustments occurring in our cluster, cadvisor provides container_cpu_cfs_throttled_periods_total and container_cpu_cfs_periods_total . With these two you can easily calculate % throttling in all CPU cycles.

Best practices

Be careful with limits and requests

Limits are a way to set maximum limits on the resources of your node, but these limits should be handled with care, as your application could end up with a throttled process or removed.

Prepare against deportation

By setting very low requirements you might think that this will give a minimum of CPU or Memory to your processes. But the kubelet will discard the Pods with higher usage than required first, so you are marking those Pods as the first to be dropped!

In case you need to protect specific Pods from being prioritized (when kube-scheduler needs to allocate a new Pods), assign Priority Classes to your most important processes.

Throttling is a silent enemy

By setting unrealistic or over-committed limits, you may not know that your processes are being throttled and performance is affected. Proactively monitor your CPU usage and know your actual limits in both containers and namespaces. An illustration of kubernetes’ CPU and memory management

Here I give an example with the Sun Spinner service of Sunteco Cloud. First, you need to choose the right resources for container applications:

The special feature of Sun Spinner is that it allows users to easily monitor the CPU and RAM parameters of the container, and reasonably adjust the resources for the pods to ensure all optimal operation.

You can experience the Sun Spinner service for yourself here .

If you have any questions regarding the article, leave a comment below. I will reply as soon as possible!