Understanding Container Scheduling in Kubernetes: Resource Allocation vs. Priority Management

Figure 1: Stress program with default priority

In above image we are running a “stress” program is running. And two instances of “stress” are running one is consuming 93.8% CPU and other is consuming 0.0% CPU.

These two processes are started with default priority i.e, 0 as show in above figure NI column.

Even if we change the priority of above process, let say we changed priority to 16.

Figure 2: Stress program with 16 priority

Post changing the priority to 16 which means we are lowering the priority than before. However, in Figure 2, our stress program is still using ~100% of one CPU! This situation seems contradictory at first glance because one might expect a lower-priority process to yield CPU resources to higher-priority processes.

The explanation lies in the nature of CPU scheduling and how priority is interpreted. Priority, in this context, is a relative measurement within the context of other processes competing for CPU time. If there are no other processes demanding CPU resources, even a process with a lower priority (higher nice value) can still fully utilize the CPU if it desires.

In simpler terms, priority doesn’t dictate an absolute limit on CPU usage; instead, it influences how CPU time is allocated among competing processes when there is contention for resources. If there are no competing processes or if they have lower priority, a lower-priority process can still consume as much CPU as it needs.

This behavior might seem counterintuitive, especially if one expects priority to directly limit CPU usage. However, it reflects the flexible and dynamic nature of CPU scheduling, where processes adjust their resource consumption based on system load and the priorities of other processes.

After all, if the CPU is available, shouldn’t we want our application components to be able to use it? This setup sounds resonable, it’s not suitable for our containerized applications for below two reason:-

1. Dynamic Resource Allocation: In a Kubernetes environment, containers should be able to be allocated to any host with enough resources to run them. It’s not practical for administrators to manually set the relative priority of every container, especially in a multi-tenant environment where multiple applications or teams are using the same cluster.
2. Resource Utilization: Without some idea of how much CPU a particular container will use, Kubernetes cannot know which hosts are full and which ones have more room available. If multiple containers on the same host become busy simultaneously, they will compete for available CPU cores, leading to performance degradation for all containers on that host.