How it was found

Description

The Amazon VPC CNI controller, when configured to manage NetworkPolicy rules, will incorrectly apply firewall rules for Completed pods as if the pods are still running, causing these rules to be applied to other unrelated pods that happen to receive the same IP address as a Completed pod.

For example, if you have a pod A with IP address X. Pod A is allowed some access via NetworkPolicy, which the VPC CNI controller effects by adding firewall rules on nodes for IP address X. When Pod A finished (for example if it's a Job) these firewall rules are not removed. As the pod is finished other pods are free to receive the same IP address. If pod B which is not granted any access by NetworkPolicy is assigned the same IP address X, then pod B will inherit the network access pod A had, until the Completed pod A is deleted.

The correct behavior would be to check for pod completion and treat pod completion the same as pod deletion, removing firewall rules when a pod is finished.

Affected versions

I tested this on EKS version 1.33:

• With auto mode with the network policy controller enabled
• Without auto mode, with VPC CNI add-on v1.19.5-eksbuild.1
• Without auto mode, with VPC CNI add-on v1.20.1-eksbuild.3 Strict mode does not have any effect.

POC

On an EKS cluster with auto mode enabled, or the VPC CNI controller with enableNetworkPolicy set to true, apply the following preparatory Kubernetes resources:

---
apiVersion: v1
kind: Namespace
metadata:
  name: networkpolicy-test
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: whoami
  namespace: networkpolicy-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: whoami
  template:
    metadata:
      labels:
        app: whoami
    spec:
      containers:
        - name: whoami
          image: traefik/whoami:latest
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: whoami
spec:
  ports:
    - port: 80
      targetPort: 80
      protocol: TCP
  selector:
    app: whoami
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: ingress
  namespace: networkpolicy-test
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          test: allowed
    ports:
    - protocol: TCP
      port: 80

This NetworkPolicy configuration will allow the following Job to connect to the Deployment

---
apiVersion: batch/v1
kind: Job
metadata:
  name: allowed
  namespace: networkpolicy-test
spec:
  template:
    metadata:
      name: curl
      labels:
        test: allowed
    spec:
      containers:
        - name: curl
          image: "alpine/curl:latest"
          command: ["curl", "-v", "whoami.networkpolicy-test.svc.cluster.local:80"]
      restartPolicy: Never

but the following Job will be blocked and time out:

---
apiVersion: batch/v1
kind: Job
metadata:
  name: blocked
  namespace: networkpolicy-test
spec:
  template:
    metadata:
      name: curl
      labels:
        test: blocked
    spec:
      containers:
        - name: curl
          image: "alpine/curl:latest"
          command: ["curl", "-v", "whoami.networkpolicy-test.svc.cluster.local:80"]
      restartPolicy: Never

However if there is an allowed job in Completed state, the blocked pod has a chance of receiving the same IP address as the completed allowed pod, when that happens the blocked pod will, in violation of the NetworkPolicy, be able to access the Deployment.

To make it easy to reproduce I would suggest applying the attached █████████ file to create 20 allowed Jobs. Wait for all 20 Jobs to finish, then apply the attached ██████████to create 20 blocked Jobs. Almost certainly some of the blocked pods will be given IP addresses matching those of completed allowed pods, and will therefore be able to connect to the Deployment and exit successfully. Some of the blocked pods will be given different IP addresses and as a result will correctly not be able to connect to the Deployment, hang and then error.

Impact

Summary: On an EKS cluster relying on Kubernetes NetworkPolicy rules to restrict network access, having Completed pods that were allowed some access will result in other pods being given the same access.