> ## Documentation Index
> Fetch the complete documentation index at: https://www.truefoundry.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Custom Networking in EKS

> Configure custom networking in EKS for TrueFoundry.

## Overview

By default, Amazon VPC CNI will assign Pods an IP address selected from the primary subnet. The primary subnet is the subnet CIDR that the primary ENI is attached to, usually the subnet of the node/host.

If the subnet CIDR is too small, the CNI may not be able to acquire enough secondary IP addresses to assign to your Pods. This is a common challenge for EKS IPv4 clusters.

Custom networking is one solution to this problem and is supported by TrueFoundry. For more details, please refer to the [Amazon EKS Custom Networking documentation](https://docs.aws.amazon.com/eks/latest/best-practices/custom-networking.html).

## Setup

To setup custom networking in EKS, you need to follow the steps below:

1. Attach a secondary CIDR to the VPC and create new subnets with the secondary CIDR in the same AZs as the primary subnet. For example, if the primary subnet is 10.0.0.0/16, you can attach a secondary CIDR of 100.64.0.0/16 ([RFC6598](https://datatracker.ietf.org/doc/html/rfc6598)) to the VPC and if you have primary subnets in 3AZs which are used for the EKS nodes, you have to create 3 new subnets with the secondary CIDR in the same AZs.
2. Ensure that secondary subnets are added to the route tables of your primary subnets. This should happen automatically when you create the subnets.
3. We need to make the AWS VPC CNI aware of the secondary subnets. If you have used TrueFoundry's OpenTofu/Terraform code to setup your cluster, you need to make the following change in the [EKS module](https://github.com/truefoundry/terraform-aws-truefoundry-cluster)

```hcl theme={"dark"}
cluster_addons_vpc_cni_additional_configurations = {
    env = {
      AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG = "true"
      ENI_CONFIG_LABEL_DEF               = "topology.kubernetes.io/zone"
    }
}
```

4. In all other cases you can set the same env variable by running the following commands from the kubernetes cli:

```bash theme={"dark"}
kubectl set env daemonset aws-node -n kube-system "AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true"
kubectl set env daemonset aws-node -n kube-system "ENI_CONFIG_LABEL_DEF=topology.kubernetes.io/zone"
```

5. Create the following `ENIConfig` resources in the cluster for each AZ of the EKS nodes

```yaml theme={"dark"}
apiVersion : crd.k8s.amazonaws.com/v1alpha1
kind : ENIConfig
metadata:
  name: us-east-1a # us-east-1a is the AZ of the EKS nodes
spec:
  securityGroups:
    - sg-0dff111a1d11c1c11 # sg-0dff111a1d11c1c11 is the security group of the EKS nodes
  subnet: subnet-011b111c1f11fdf11 # subnet-011b111c1f11fdf11 is the subnet with the secondary CIDR
```

6. Restart the nodes one by one to apply the changes in the cluster networking. Pods will be rescheduled to the new nodes with the secondary IP addresses.