Skip to main content

Headroom

One of Ocean’s key features for optimizing scaling is headroom, a buffer of spare capacity ensuring that a cluster is always ready for a rapid application scale-up.

When you configure headroom in specific amounts of resources (vCPU, memory, and GPU) or specify headroom as a percentage of the cluster’s total requested resources, the cluster can scale workloads without waiting for new instances to be provisioned.

Ocean optimally manages the headroom dynamically to provide the best possible cost-performance balance. You can also configure headroom manually to meet specific requirements.

How It Works​

Ocean continually checks for the cluster's unscheduled workloads (pods or tasks). If unscheduled workloads are found, Ocean simulates placing them on the existing infrastructure.

Since Ocean has saved compute resources in advance (i.e., created headroom), the container orchestrator can instantly schedule unscheduled workloads for these nodes. Once the headroom has been used to scale up workloads, Ocean will provide more infrastructure to balance the headroom amount in preparation for the next service scaling up.

You can configure Headroom using either an automatic or manual mechanism.

Automatic Headroom​

Automatic refers to dynamic headroom designed to predict the next scale-up of services in the cluster. It is limited by a percentage of the overall resources requested by the cluster’s workloads.

When configuring automatic headroom, Ocean calculates the total amount of resources (vCPU, memory, and GPU) deployed in the cluster and allocates a specific percentage for availability as spare capacity. The default is 5%, and is configurable.

Ocean continually monitors the cluster and creates headroom to prepare the cluster for the next workload scale-out:

  1. Ocean analyzes historical workload behavior and identifies which workloads are most likely to scale out and by how much.
  2. Ocean provisions infrastructure to accommodate the top 5 workloads identified in step 1, considering all their constraints (such as resource requests and affinity rules).
  3. Ocean maintains the total headroom to match the user-defined percentage of cluster resources.

As these steps are repeated, the required headroom percentage is always maintained.

Automatic Headroom per Virtual Node Group​

You can define automatic headroom per virtual node group. The calculation of the automatic headroom in the virtual node group is the same as the calculation for the cluster (as described above).

Since cluster-level and virtual node group-level automatic headroom are calculated independently, enabling both will result in duplicate headroom allocation for the same workloads. To avoid over-provisioning, configure automatic headroom either at the cluster level or at the VNG level, but not both.

note

This option does not apply to ECS. In ECS, the automatic headroom is calculated based on the service with the most changed tasks in the cluster, regardless of the virtual node group settings.

Manual Headroom​

While automatic headroom is the recommended configuration, you can also set up manual headroom on top of automatic headroom. Defined per virtual node group, manual headroom adds flexibility to help with specific scenarios.

Manual headroom is configured as specific resource amounts and is useful when you know your cluster's scaling patterns and exact resource requirements in advance.

Manual headroom involves setting headroom units—i.e., chunks of resources such as vCPU, memory, and GPU—to be provisioned and saved for future scale-out.

You can define manual headroom at the Virtual Node Group level to set those units for different workloads with different constraints. This makes sure that the provisioned spare capacity will exist on nodes with the right configuration to run the scaling workloads.

For example, if two virtual node groups are each configured to maintain two headroom units of 2048 MiB and 2000 CPU, Ocean will maintain a total of four headroom units—two for each virtual node group.

Note for AWS and GKE Users​

Due to backward compatibility considerations, you will need to activate a switch in the configuration in order to enable automatic and manual headroom to work in parallel. Otherwise, automatic headroom overrides all other manually configured headroom definitions, whether at cluster or virtual node group level. You may enable automatic and manual headroom at the Ocean level by setting autoScaler.enableAutomaticAndManualHeadroom to True.