Compute Resources in Tekton

Background: Resource Requirements in Kubernetes

Kubernetes allows users to specify CPU, memory, and ephemeral storage constraints for containers. Resource requests determine the resources reserved for a pod when it’s scheduled, and affect likelihood of pod eviction. Resource limits constrain the maximum amount of a resource a container can use. A container that exceeds its memory limits will be killed, and a container that exceeds its CPU limits will be throttled.

A pod’s effective resource requests and limits are the higher of:

  • the sum of all app containers request/limit for a resource
  • the effective init container request/limit for a resource

This formula exists because Kubernetes runs init containers sequentially and app containers in parallel. (There is no distinction made between app containers and sidecar containers in Kubernetes; a sidecar is used in the following example to illustrate this.)

For example, consider a pod with the following containers:

Container CPU request CPU limit
init container 1 1 2
init container 2 2 3
app container 1 1 2
app container 2 2 3
sidecar container 1 3 no limit

The sum of all app container CPU requests is 6 (including the sidecar container), which is greater than the maximum init container CPU request (2). Therefore, the pod’s effective CPU request will be 6.

Since the sidecar container has no CPU limit, this is treated as the highest CPU limit. Therefore, the pod will have no effective CPU limit.

Task Resource Requirements

Tekton allows users to specify resource requirements of Steps, which run sequentially. However, the pod’s effective resource requirements are still the sum of its containers’ resource requirements. This means that when specifying resource requirements for Step containers, they must be treated as if they are running in parallel.

Tekton adjusts Step resource requirements to comply with LimitRanges. ResourceQuotas are not currently supported.

LimitRange Support

Kubernetes allows users to configure LimitRanges, which constrain compute resources of pods, containers, or PVCs running in the same namespace.

LimitRanges can:

  • Enforce minimum and maximum compute resources usage per Pod or Container in a namespace.
  • Enforce minimum and maximum storage request per PersistentVolumeClaim in a namespace.
  • Enforce a ratio between request and limit for a resource in a namespace.
  • Set default request/limit for compute resources in a namespace and automatically inject them to Containers at runtime.

Tekton applies the resource requirements specified by users directly to the containers in a Task's pod, unless there is a LimitRange present in the namespace. (Tekton doesn’t allow users to configure init containers for a Task.) Tekton supports LimitRange minimum, maximum, and default resource requirements for containers, but does not support LimitRange ratios between requests and limits (#4230). LimitRange types other than “Container” are not considered for purposes of resource requirements.

Requests

If a container does not have requests defined, the resulting container’s requests are the larger of:

  • the LimitRange minimum resource requests
  • the LimitRange default resource requests (for init and Sidecar containers) or the LimitRange default resource requests divided by the number of Step containers (for Step containers)

If a container has requests defined, the resulting container’s requests are the larger of:

  • the container’s requests
  • the LimitRange minimum resource requests

Limits

If a container does not have limits defined, the resulting container’s limits are the smaller of:

  • the LimitRange maximum resource limits
  • the LimitRange default resource limits

If a container has limits defined, the resulting container’s limits are the smaller of:

  • the container’s limits
  • the LimitRange maximum resource limits

Examples

Consider the following LimitRange:

apiVersion: v1
kind: LimitRange
metadata:
  name: limitrange-example
spec:
  limits:
  - default:  # The default limits
      cpu: 2
    defaultRequest:  # The default requests
      cpu: 1
    max:  # The maximum limits
      cpu: 3
    min:  # The minimum requests
      cpu: 300m
    type: Container

A Task with 2 Steps and no resources specified would result in a pod with the following containers:

Container CPU request CPU limit
container 1 500m 2
container 2 500m 2

Here, the default CPU request was divided among the step containers, and this value was used since it was greater than the minimum request specified by the LimitRange. The CPU limits are 2 for each container, as this is the default limit specifed in the LimitRange.

If we had a Task with 2 Steps and 1 Sidecar with no resources specified would result in a pod with the following containers:

Container CPU request CPU limit
container 1 500m 2
container 2 500m 2
container 3 1 2

For the first two containers, the default CPU request was divided among the step containers, and this value was used since it was greater than the minimum request specified by the LimitRange. The third container is a sidecar and since it is not a step container gets the full default CPU request of 1. AS before the CPU limits are 2 for each container, as this is the default limit specifed in the LimitRange.

Now, consider a Task with the following Steps:

Step CPU request CPU limit
step 1 200m 2
step 2 1 4

The resulting pod would have the following containers:

Container CPU request CPU limit
container 1 300m 2
container 2 1 3

Here, the first Step's request was less than the LimitRange minimum, so the output request is the minimum (300m). The second Step's request is unchanged. The first Step's limit is less than the maximum, so it is unchanged, while the second Step's limit is greater than the maximum, so the maximum (3) is used.

Support for multiple LimitRanges

Tekton supports running TaskRuns in namespaces with multiple LimitRanges. For a given resource, the minumum used will be the largest of any of the LimitRanges’ minimum values, and the maximum used will be the smallest of any of the LimitRanges’ maximum values.

The minimum resource requirement used will be the largest of any minimum for that resource, and the maximum resource requirement will be the smallest of any of the maximum values defined. The default value will be the minimum of any default values defined. If the resulting default value is less than the resulting minimum value, the default value will be the minimum value.

It’s possible for multiple LimitRanges to be defined which are not compatible with each other, preventing pods from being scheduled.

Example

Consider a namespaces with the following LimitRanges defined:

apiVersion: v1
kind: LimitRange
metadata:
  name: limitrange-1
spec:
  limits:
  - default:  # The default limits
      cpu: 2
    defaultRequest:  # The default requests
      cpu: 750m
    max:  # The maximum limits
      cpu: 3
    min:  # The minimum requests
      cpu: 500m
    type: Container

apiVersion: v1
kind: LimitRange
metadata:
  name: limitrange-2
spec:
  limits:
  - default:  # The default limits
      cpu: 1.5
    defaultRequest:  # The default requests
      cpu: 1
    max:  # The maximum limits
      cpu: 2.5
    min:  # The minimum requests
      cpu: 300m
    type: Container

A namespace with limitrange-1 and limitrange-2 would be treated as if it contained only the following LimitRange:

apiVersion: v1
kind: LimitRange
metadata:
  name: aggregate-limitrange
spec:
  limits:
  - default:  # The default limits
      cpu: 1.5
    defaultRequest:  # The default requests
      cpu: 750m
    max:  # The maximum limits
      cpu: 2.5
    min:  # The minimum requests
      cpu: 300m
    type: Container

Here, the minimum of the “max” values is the output “max” value, and likewise for “default” and “defaultRequest”. The maximum of the “min” values is the output “min” value.

ResourceQuota Support

Kubernetes allows users to define ResourceQuotas, which restrict the maximum resource requests and limits of all pods running in a namespace. TaskRuns can’t currently be created in a namespace with ResourceQuotas without siginificant caveats. (#2933).

References