asrin andirin // devops & cloud infrastructure

Image Pulls Are a Disk Workload: Tuning gp3 and Kubelet Pull Limits for High-Density EKS Nodes

Some nodes pull far more images than others. A single 16xlarge or 24xlarge node can run 200 pods or more, and in a cluster where Karpenter consolidates often, or where a drift replaces many nodes at once, a new node may receive 50 to 100 pods within a short time. Every new node starts with an empty image cache, so each of those pods has to pull its images before it can start.

The numbers in this article come from the platform I run: Kubernetes 1.35, around 500 nodes, mostly a mix of 4xlarge and 8xlarge instances from the m families.

During these node waves, some pods stay in ContainerCreating or ImagePullBackOff much longer than expected, which delays application startups. When I first saw this, I assumed we were hitting ECR’s rate limits. That was wrong. The first cause was the kubelet’s own rate limit for image pulls, which shows up in the pod events as pull QPS exceeded. The second was the root volume, and that one produces no error at all. The pulls are just slow.

The two are connected. The kubelet’s limit decides how many pulls run at the same time, and every one of those pulls writes to the same disk. Raising the limit removed the error and exposed the disk as the real bottleneck. After also tuning the disk, the share of image pulls taking longer than 3 minutes dropped from 11% to zero, and the slowest pull went from over 6 minutes to about 2, in the measurements below.

What an Image Pull Does to the Disk

Pulling an image is more than a download. containerd fetches the compressed layers and then unpacks them into the snapshotter under /var/lib/containerd. Unpacking means decompressing and writing every file in every layer. The uncompressed data is bigger than the image size you see in the pod event, and it lands as lots of small files, which costs IOPS as well as throughput.

On EKS that directory sits on the root EBS volume. A gp3 volume gives you 3,000 IOPS and 125 MiB/s by default, whatever its size. That is enough for a node pulling a few images at a time, but not for one that gets dozens of pods in the same minute, each unpacking hundreds of megabytes.

Why Dense, Fast-Scaling Nodes Hit It Hardest

The load on the disk depends on how many pods one node receives at once, and that grows with the instance size. A node with room for 20 pods rarely has more than a few pulls running together. A node with room for 200 pods can easily get a hundred pull requests in the first seconds after it joins.

To see how many pods your nodes can hold:

kubectl get nodes -o custom-columns='NAME:.metadata.name,TYPE:.metadata.labels.node\.kubernetes\.io/instance-type,PODS:.status.capacity.pods'

The defaults per instance type are listed in Amazon EKS recommended maximum Pods for each Amazon EC2 instance type. With VPC CNI prefix delegation the numbers go much higher; this article explains why.

Scaling down matters as much as scaling up. The image cache lives on the node, so it goes away with the node. Every consolidation, drift or spot interruption moves pods to a node that may have none of their images, and a cluster that scales up and down often goes through that cold start again and again.

The Kubelet Decides How Many Pulls Hit the Disk at Once

The kubelet puts every image pull through a token bucket. Kubernetes’ default values:

ParameterDefaultDescription
registryPullQPS5Average number of image pull requests allowed per second
registryBurst10Extra capacity allowed for short bursts of requests

registryPullQPS is the refill rate and registryBurst is the size of the bucket. With the defaults, 10 pulls can go through at once, and then the bucket refills at 5 per second. These values apply per node. They do not mean the whole cluster can only pull 5 images per second.

When the bucket is empty, the kubelet doesn’t queue the request. It fails the pull on the spot with pull QPS exceeded, the pod goes to ErrImagePull and then ImagePullBackOff, and the retry waits on an exponential backoff that starts at 10 seconds and goes up to 5 minutes. Lose that race a few times in a row and a pod can wait minutes for an image the node could have pulled in seconds.

I first read this error as an Amazon ECR rate limit, which is what the wording suggests. It isn’t one. The limit comes from the kubelet’s configuration on the node, not from the image registry.

The limit counts pull requests, not pods. Init containers and sidecars take their own tokens, so a pod with an app container, two sidecars and an init container uses four. With imagePullPolicy: Always, the kubelet sends a pull request on every container start, even when the image is already on the node, and that costs a token as well. registryBurst only matters when registryPullQPS is above 0. Setting registryPullQPS: 0 turns the limit off.

Once a pull has a token, nothing else holds it back. The EKS AMI runs with serializeImagePulls: false and no maxParallelImagePulls, so every accepted pull starts immediately and they all write to the root volume together. That makes the token bucket the only thing between a node wave and the disk. With the defaults, rejections and backoffs spread the pulls out over time, and that partly hides how slow the disk is.

I looked at maxParallelImagePulls as a way to control this and decided against it. It caps how many pulls run at once, so the first pods start sooner, but it has no idea which pods matter. A monitoring sidecar can get a slot before your API pod does.

The current values on a node:

kubectl get --raw "/api/v1/nodes/<node-name>/proxy/configz" \
  | jq '.kubeletconfig | {registryPullQPS, registryBurst, serializeImagePulls, maxParallelImagePulls}'

And recent pull QPS exceeded events:

kubectl get events -A -o json \
  | jq -r '.items[] | select((.message // "") | test("pull QPS")) | "\(.lastTimestamp) \(.involvedObject.namespace)/\(.involvedObject.name)"'

Events only live for about an hour by default, so run this right after a scale-up or a node rotation. An empty result on a quiet afternoon doesn’t mean you’re safe.

Raising the Pull Limit

The EC2NodeClass has a spec.kubelet block for things like maxPods, reserved resources and eviction thresholds. registryPullQPS and registryBurst aren’t in it, and there’s no way to pass arbitrary kubelet fields through it.

On AL2023 the way around this is userData. AL2023 nodes boot with nodeadm, which reads a NodeConfig object. Karpenter builds its own NodeConfig for each node and merges yours into it, so you only write the fields you want to change:

apiVersion: karpenter.k8s.aws/v1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2023
  # subnets, security groups, role, blockDeviceMappings ...
  userData: |
    apiVersion: node.eks.aws/v1alpha1
    kind: NodeConfig
    spec:
      kubelet:
        config:
          registryPullQPS: 25
          registryBurst: 50

Anything under spec.kubelet.config goes to the kubelet as a KubeletConfiguration field, so the same trick works for other kubelet settings Karpenter doesn’t expose.

One thing that caught me: EKS managed node groups don’t get any of this. A managed node group running next to Karpenter keeps the defaults until you add the same NodeConfig to its launch template user data, as a MIME multi-part document.

Choosing the Values

There is no universal number here. The goal is to give the kubelet enough room for the largest burst a single node sees.

The default burst of 10 gives you a way to measure it. During a node wave, the pod events on one node show two things in the same window: pulls that started and pulls rejected with pull QPS exceeded. Add them up and you get the real demand. On one of these nodes, 10 pulls were accepted and 14 rejected within the same burst, so roughly 25 concurrent requests. registryPullQPS: 25 and registryBurst: 50 cover that peak with room to spare.

25 and 50 are the only values I ran, on 4xlarge and 8xlarge nodes, so I don’t have tested numbers for other sizes. The counting method works on any node, though, and the answer depends on the workload as much as the instance: a node full of small pods sharing one base image needs less than a node running pods with several large, unique images.

You can also set registryPullQPS: 0 and drop the limit entirely. I wouldn’t do that in production. With hundreds of nodes starting at once, the next thing you hit is your account’s registry API quota, and that one affects every node, not just one.

Verifying

When the first replacement node is Ready, check that the kubelet picked up the new values with the same configz command. If you still see 5 and 10, the NodeConfig didn’t make it to the kubelet, and the AMI family is the first thing to check.

New values in configz don’t prove much on their own. What you want to know is whether pull QPS exceeded still happens on the new nodes. In this cluster, every error after the change came from a node still on 5 and 10: nodes waiting for their turn in the rollout, nodes from other classes, and the managed node group. None came from a node running 25 and 50.

At that point I considered the problem solved, until I looked at how long the pulls were taking.

Measuring Pull Times

The kubelet logs how long each successful pull took:

Successfully pulled image "<image>" in 4m51.528s (4m51.528s including waiting). Image size: 551588570 bytes.

The message has two durations. The first is the pull itself, the second adds the time the request waited inside the kubelet before it started. When both are the same, nothing was waiting in the kubelet. The pull started right away and was just slow.

On the new nodes, more than 40% of pulls took over a minute, and 11% took over three. The duration had almost nothing to do with the image size: a 61 MB image took 4m 20s and a 1.5 GB image took 4m 35s. Most of the slowest pulls, whatever their size, finished within the same couple of minutes.

That points to something shared on the node. The pulls all started together and finished when the whole batch was done, so the small images were stuck behind the large ones. A higher QPS limit means more pulls run on the same volume at once, and the volume was still a default gp3.

If you run node-exporter, you can watch the disk during a wave:

# Disk busy time. Close to 1 means the device is saturated.
rate(node_disk_io_time_seconds_total{instance=~"<node-ip>:.*", device="nvme0n1"}[1m])

# Write throughput in MiB/s. Flat at ~125 means the gp3 throughput limit.
rate(node_disk_written_bytes_total{instance=~"<node-ip>:.*", device="nvme0n1"}[1m]) / 1048576

# Total IOPS. Flat at ~3000 means the gp3 IOPS limit.
sum(
  rate(node_disk_reads_completed_total{instance=~"<node-ip>:.*", device="nvme0n1"}[1m])
  + rate(node_disk_writes_completed_total{instance=~"<node-ip>:.*", device="nvme0n1"}[1m])
)

On my nodes the IOPS graph was the clear one. Right after a node came up, IOPS climbed to the gp3 limit and stayed there as a flat line while the pulls were running.

A before and after test on the disk settings is still the most convincing check. The script I used for it is at the end of the article.

Tuning the Disk

gp3 lets you set IOPS and throughput separately from the volume size. In Karpenter both go under blockDeviceMappings:

spec:
  blockDeviceMappings:
  - deviceName: /dev/xvda
    ebs:
      volumeSize: 100Gi
      volumeType: gp3
      iops: 6000        # default 3000
      throughput: 500   # MiB/s, default 125
      encrypted: true
      deleteOnTermination: true

gp3 allows at most 0.25 MiB/s of throughput per provisioned IOPS and at most 500 IOPS per GiB, so 500 MiB/s needs at least 2,000 IOPS and 6,000 IOPS needs a volume of at least 12 GiB. A root volume clears both easily.

Results

Same NodePool, same workloads, measured during a rollout that refilled every node. I only counted nodes created with the new disk settings, and only after all their pods were running, so slow pulls still in progress couldn’t be left out.

Pull durationgp3 default (3000 / 125)gp3 tuned (6000 / 500)
Under 10 seconds29%41%
10 to 60 seconds29%48%
1 to 3 minutes31%10%
Over 3 minutes11%0%
Slowest pull6m 21s2m 07s

Some of the same images on both runs: a 334 MB image went from 4m 44s to 1m 28s, a 592 MB image from 6m 21s to 1m 51s, and the 1.5 GB image from 4m 35s to 2m 07s.

Node waves differ from run to run, so this is field data rather than a benchmark. The shift is large enough that I’m confident the root volume was the bottleneck.

I raised IOPS and throughput in the same change and never tested them separately, so I can’t say how much of the gain came from each.

At us-east-1 prices, going from 3,000 to 6,000 IOPS and from 125 to 500 MiB/s adds about $30 per node per month.

Instance EBS Limits

More IOPS and throughput only help up to the instance’s own EBS bandwidth. The node gets whichever is lower:

effective throughput = min(gp3 provisioned throughput, instance EBS bandwidth)

Every instance type has a baseline EBS bandwidth it can sustain, and many smaller sizes can burst higher for about 30 minutes in a 24-hour period. A node wave fits in that window, but size for the baseline. Anything above it is throughput you pay for and can’t use.

To check the instance types in your NodePools:

aws ec2 describe-instance-types \
  --instance-types c6i.4xlarge m6i.8xlarge c5a.4xlarge \
  --query 'InstanceTypes[].[InstanceType,EbsInfo.EbsOptimizedInfo.BaselineThroughputInMBps,EbsInfo.EbsOptimizedInfo.MaximumThroughputInMBps,EbsInfo.EbsOptimizedInfo.BaselineIops,EbsInfo.EbsOptimizedInfo.MaximumIops]' \
  --output table

The columns are baseline MB/s, burst MB/s, baseline IOPS and burst IOPS. The full list is in Amazon EBS-optimized instance types.

In practice, the 4xlarge and 8xlarge sizes of current generations sustain several times the gp3 default and gain the most from tuning. Small sizes often can’t use 500 MiB/s at all. Some older families are noticeably slower than newer ones of the same size. A c5a.4xlarge sustains about 200 MB/s, while a c6i.4xlarge sustains 625 MB/s. The 5th-generation AMD families (c5a, m5a, r5a and their d variants) are the ones I removed from the application NodePools. Karpenter only drifts the nodes currently running on the removed types, so that change is cheap.

Instance Store

Types with a d suffix come with local NVMe drives that are much faster than EBS. With instanceStorePolicy: RAID0 on the EC2NodeClass, Karpenter puts the kubelet and containerd data on them and EBS drops out of the pull path:

spec:
  instanceStorePolicy: RAID0

I priced it before going with gp3. In a spot price snapshot from one Availability Zone, the d variant cost 46% more for c5.4xlarge, 41% more for m5.4xlarge, 25% more for c6i.4xlarge, and 15 to 17% more for the 8xlarge sizes I checked. On most pairs that’s two to three times the cost of the gp3 tuning. The exception was m6id.4xlarge, priced the same as m6i.4xlarge at that moment, which is worth a look if you’re considering this.

Spot capacity is the other cost. A NodePool limited to d types has fewer spot pools to pick from, so expect more interruptions and more insufficient capacity errors. For me gp3 got most of the gain for a fraction of the price. I’d only move to instance store if gp3 at the instance baseline still isn’t enough.

Smaller Images, Fewer Waves

On a saturated volume, pull duration stopped following image size: the 61 MB image took as long as the 1.5 GB one, because all of them shared the same disk. So a large image slows down every pod that lands on the node in the same wave, not only its own. Many of the slowest images in this cluster were 500 MB and up. Slimming them down helps on every node, and unlike the disk tuning it adds nothing to the bill.

The number of waves matters too, since each one is a cold start for the nodes involved. Aggressive consolidation, a short consolidateAfter with a big disruption budget, creates more waves than the workload needs.

Placeholder pods also help, because a scale-up then lands on existing nodes that often have the images cached already; see Karpenter Overprovisioning.

Rolling Out

Both userData and blockDeviceMappings are part of the EC2NodeClass spec. Change either one and Karpenter marks every node of that class as drifted, then replaces them within the NodePool disruption budgets. One line in a shared class can rotate a big part of the cluster. If you’re making both changes, ship them together and rotate once.

Before applying, look at which NodePools reference the class, because all of them will rotate. Check the disruption budgets too. At 10%, a large NodePool takes hours to roll over. The rollout is also a node wave in itself, with every rescheduled pod pulling images on a fresh node, which makes it a good moment to measure.

If the same class also runs Prometheus, Loki ingesters or other stateful system components, they get restarted too, for nothing. Their images are pulled once and they don’t see node waves. A separate EC2NodeClass for system NodePools avoids that. Copy the current class with the same subnets, security groups, role and AMI under a new name, then point the system NodePools at it:

# in each system NodePool
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: system

Apply the new EC2NodeClass before you point anything at it, because a NodePool that references a missing class can’t launch nodes. Check that all its conditions are True first:

kubectl get ec2nodeclass system -o json \
  | jq -r '.status.conditions[] | "\(.type)=\(.status) \(.message // "")"'

Moving a NodePool to another class is a drift too, so the system pools rotate once. After that, changes to the application class leave them alone.

The Measurement Script

This is the script behind the numbers above. It finds the nodes created with the current EC2NodeClass hash, collects the Pulled events for the pods on them, and prints a distribution plus the slowest pulls.

NODECLASS=<nodeclass-name>
HASH=$(kubectl get ec2nodeclass "$NODECLASS" \
  -o jsonpath='{.metadata.annotations.karpenter\.k8s\.aws/ec2nodeclass-hash}')

NODES=$(kubectl get nodeclaims -l karpenter.k8s.aws/ec2nodeclass="$NODECLASS" -o json \
  | jq -r --arg h "$HASH" '.items[] | select(.metadata.annotations["karpenter.k8s.aws/ec2nodeclass-hash"]==$h) | .status.nodeName // empty')

echo "=== New nodes and pods not yet running (0 = settled)"
echo "$NODES" | while read n; do
  [ -n "$n" ] && echo "$n pending=$(kubectl get pods -A --field-selector spec.nodeName="$n",status.phase!=Running,status.phase!=Succeeded --no-headers 2>/dev/null | wc -l | tr -d ' ')"
done

PODS=$(echo "$NODES" | while read n; do
  [ -n "$n" ] && kubectl get pods -A --field-selector spec.nodeName="$n" \
    -o jsonpath='{range .items[*]}{.metadata.namespace}/{.metadata.name}{"\n"}{end}'
done)

RES=$(kubectl get events -A --field-selector reason=Pulled -o json | jq -r '
  def secs: if test("ms$") then (sub("ms$";"")|tonumber/1000)
            elif test("µs$") then 0
            else (capture("^((?<m>[0-9]+)m)?(?<s>[0-9.]+)s$") | ((.m // "0")|tonumber)*60 + (.s|tonumber)) end;
  .items[] | select((.message // "") | test("Successfully pulled"))
  | (.message | capture("in (?<t>[^ ]+) \\((?<w>[^ ]+) including waiting\\)\\. Image size: (?<s>[0-9]+)")) as $m
  | ($m.t|secs) as $sec | (($m.s|tonumber)/1048576) as $mb
  | "\(.involvedObject.namespace)/\(.involvedObject.name)\t\($sec|floor)\t\($mb|floor)MB\t\(if $sec>0 then (($mb/$sec*10)|floor)/10 else 0 end)MB/s"' \
  | grep -F -f <(echo "$PODS"))

echo "=== Distribution"
echo "$RES" | awk -F'\t' '{s=$2+0; if(s<10)a++; else if(s<60)b++; else if(s<180)c++; else d++}
  END{printf "<10s: %d  10-60s: %d  1-3m: %d  >3m: %d  total: %d\n", a, b, c, d, NR}'

echo "=== Slowest 15 pulls (seconds / size / speed)"
echo "$RES" | sort -t$'\t' -k2 -n | tail -15

Run it once pending is 0 on the nodes you care about. It only sees finished pulls, so a node that’s still pulling makes the numbers look better than they are. Run it soon after the wave as well, since events expire after about an hour.

The secs function is there for a reason. Durations come as Go strings like 850ms, 12.3s or 4m51.5s. My first version sorted them as text, 4m51s landed before 8s, the slowest pulls never showed up, and the results looked much better than they were.

Conclusion

On nodes that run hundreds of pods and get replaced often, image pulls put a heavy write load on the root volume. Every wave starts from an empty cache and unpacks all of the node’s images onto one root volume, and a default gp3 volume can’t keep up with that.

The kubelet’s pull limit and the disk have to be sized together. With the default limit you see pull QPS exceeded, and the backoffs hide the disk. Raise the limit and the error goes away, but the pulls all land on the volume at once. After the first fix, measure the pull durations. If small images take as long as big ones, look at the disk.


Official Documentation