VPC-ENI Specified Subnet IP Allocation (Container Network v2)

Overview

The VPC-ENI container network mode supports assigning subnets for pod IP allocation. Users can use this feature to plan and allocate IP addresses from different subnets for pods serving various business uses.

Note: To enable this feature, please contact Baidu AI Cloud customer service to activate the allow list.

Requirement scenarios

Requirement scenario 1: Dynamic IP allocation for specified subnet

Dynamically assign IP to pods from the subnet associated with the subnet topology constraint policy matching with pod. All pods matching the subnet topology constraint policy (psts) will only be assigned IP addresses from subnets included in the policy.

Scenario 2: Manual IP allocation

The manual IP allocation policy involves providing an IP list for CCE to assign IPs to pods. After a pod is deleted, the IP address is retained. Before the IP address expires, the same IP address will be reused if the pod is recreated or migrated. By default, an IP address is retained for 7 days after a pod is deleted.

Scenario 3: Fixed IP

Fixed IP policy means providing an IP list from which CCE assigns IP to stateful workloads pods. After Pod deletion, the IP address is retained. The IP address remains unchanged after Pod reconstruction or migration. Regardless of whether the pod is deleted, the allocated IP address will not be released and only be reclaimed until the workload is deleted.

Solution overview

As demonstrated in the CRD data structure below, CCE provides a custom CRD to specify pod subnet topology spread policies for implementing subnet IP allocation in the K8S Cluster CCE.

Introduction to key data structure

Pod Subnet Topology Spread (PodSubnetTopologySpread)

The subnet topology spread object is the core working object for specifying IP allocation in subnets. Its core data structure is defined as follows:

1apiVersion: cce.baidubce.com/v2
2kind: PodSubnetTopologySpread
3metadata:
4  name: example-subnet-topology
5  namespace: default
6spec:
7# Topology Spread Object Name
8  name: example-subnet-topology
9# Among multiple subnet topology constraints, higher numerical values indicate higher priority. Default value: 0
10  priority: 0
11  subnets:
12#  must be a subnet ID in the same VPC as the current cluster, in the format of sbn-*, e.g., sbn-ccfud13pwcqf
13# When a dedicated subnet is used, users must confirm that the subnet is only used by the current K8S Cluster CCE.
14    sbn-ccfud13pwcqf: []
15  strategy:
16    releaseStrategy: TTL
17    ttl: 168h0m0s
18    type: Elastic
19# Select the pods to apply this subnet topology spread
20  selector:
21    matchLabels:
22      app: foo

The core fields of this object are as follows:

Usage restrictions

1. This function requires the ENI cross-subnet IP allocation capability of VPC. Please submit a ticket to request activation for the ENI cross-subnet IP allocation feature.
2. Pods in the kube-system namespace cannot use the designated subnet IP allocation function.
3. When using ipRange function, ensure that the specified IP range excludes special addresses like IPv4 network, gateway, broadcast, and multicast addresses. Including these special addresses may cause problem with IP allocation.
4. Pods in specified subnets can only be scheduled to nodes located in the same availability zone as the subnet. Please ensure that the AZ contains nodes in Ready status.
5. In scenarios involving fixed IPs and IP reuse, only dedicated subnets (subnets exclusively reserved for a single K8S Cluster CCE) can be used. Dedicated subnets cannot be converted to general-purpose subnets or removed from the cluster. For details, please refer to the description of dedicated subnets.
6. This function is applicable only to clusters using container network v2.

Dedicated subnet: When the user needs to assign several IPs under the specified subnet to pods, the subnet of the IP will automatically be marked as manual IP allocation mode. Subnets in manual IP allocation mode have the following characteristics:

1. Dedicated subnets must be exclusive to the current K8S Cluster CCE. CCE automatically adds exclusive tags to subnets to prevent other K8S Cluster CCEs from using them. (However, users can operate other Baidu AI Cloud products to utilize such subnets.)
2. Dedicated subnets only support manual IP allocation and do not allow automatic allocation. Users must manage IP address planning and assignments themselves.
3. The relationship between IPs and pods in dedicated subnets supports two modes: priority allocation and fixed binding. The fixed binding policy uses the pod's name as the identifier, ensuring that pods with the same name always have the same IP address.
4. The default subnet for cluster pod cannot be a dedicated subnet; otherwise, other pods in the cluster may fail to allocate IP normally
5. It does not support converting dedicated subnets to regular subnets or deleting them from the cluster

Configuration steps

Environment preparation

Create private subnet

In the Baidu AI Cloud VPC Console -> Subnet tab, create a new subnet for your VPC and save the subnet ID (subnet IDs are named in the format sbn-xxx). Note: When creating a subnet, select an availability zone associated with the K8S Cluster CCE nodes; otherwise, scheduling may fail.

Description: To enable ENI cross-subnet IP allocation function, please submit a ticket request.

Allocate IP for specified subnets in CCE

Scenario 1: Dynamic IP allocation for specified subnets

Dynamically assign IP to pods from the subnet associated with the subnet topology constraint policy matching with pod. All pods matching the subnet topology constraint policy (psts) will only be assigned IP addresses from subnets included in the policy. Applicable scenarios:

• Perform traffic statistics by subnet dimension
• Implement security policies at the subnet dimension, such as ACL rule control
• Enable Internet access for specific subnets via NAT (Network Address Translation)

1. Create psts

1apiVersion: cce.baidubce.com/v2
2# Pod topology spread table
3kind: PodSubnetTopologySpread
4metadata:
5  name: default
6  namespace: default
7spec:
8# Among multiple subnet topology constraints, the earlier the order is, the higher the priority will be.#
9  priority: 0
10  name: default-psts
11  strategy:
12    releaseStrategy: TTL
13    type: Elastic
14  subnets:
15#  must be a subnet ID in the same VPC as the current cluster, in the format of sbn-*, e.g., sbn-ccfud13pwcqf
16    sbn-ccfud13pwcqf: []
17    sbn-e8rk4zxn2ys6: []
18# Select the pods to use this subnet topology spread; if empty, all pods will use this subnet topology spread
19    selector:
20      matchLabels:
21        app: foo

2. Create workloads

1apiVersion: apps/v1
2kind: Deployment
3metadata:
4  name: elastic-deploy
5  namespace: default
6spec:
7  replicas: 1
8  selector:
9    matchLabels:
10      app: foo
11  template:
12    metadata:
13      labels:
14        app: foo
15    spec:
16      containers:
17      - image: nginx
18        name: nginx

3. Verify IP allocation results

1# kubectl get pod {podName} -oyaml
2apiVersion: v1
3kind: Pod
4metadata:
5  annotations:
6    cce.baidubce.com/PodSubnetTopologySpread: example-subnet-topology
7  generateName: elastic-deploy-56dc49b486-
8  labels:
9    app: foo
10  name: elastic-deploy-56dc49b486-d6z7b
11  namespace: default
12spec:
13  affinity:
14    nodeAffinity:
15      requiredDuringSchedulingIgnoredDuringExecution:
16        nodeSelectorTerms:
17        - matchExpressions:
18          - key: topology.kubernetes.io/zone
19            operator: In
20            values:
21            - zoneF
22  containers:
23  - image: nginx
24    imagePullPolicy: IfNotPresent
25    name: nginx
26    resources:
27      limits:
28        cce.baidubce.com/ip: "1"
29      requests:
30        cce.baidubce.com/ip: "1"

Scenario 2: Manual IP allocation

The manual IP allocation policy involves providing an IP list for CCE to assign IPs to pods. After a pod is deleted, the IP address is retained. Before the IP address expires, the same IP address will be reused if the pod is recreated or migrated. By default, an IP address is retained for 7 days after a pod is deleted.

Applicable scenarios:

• Fixed pod IP, requiring the IP to remain unchanged after Pod migration
• Use dedicated subnets and fully manage the IP of the subnet manually
• Pod names remain unchanged after multiple reconstruction, such as pods created by stateful workloads (apps/v1 StatefulSet)

1. Create psts

1apiVersion: cce.baidubce.com/v2
2kind: PodSubnetTopologySpread
3metadata:
4  name: example-subnet-topology
5  namespace: default
6spec:
7  priority: 0
8  subnets:
9    sbn-6mrkdcsyzpaw:
10# Optional. Fixed IP range; if left blank, it indicates that the default IP range of the subnet is used
11    - family: 4
12      range:
13      - start: 10.0.0.2
14        end: 10.0.0.254
15  strategy:
16    releaseStrategy: TTL
17    ttl: 168h0m0s
18    type: Elastic
19# Mandatory: Enable IP reuse
20    enableReuseIPAddress: true   
21  selector:
22    matchLabels:
23      workloadType: sts
24      fixedIP: "true"
25      app: fixedIPApp

2. Create workloads

1apiVersion: apps/v1
2# Stateful workloads must be used.
3kind: StatefulSet
4metadata:
5  name: foo
6  namespace: default
7spec:
8  replicas: 1
9  selector:
10    matchLabels:
11      app: fixedIPApp
12  serviceName: foo
13  template:
14    metadata:
15      labels:
16        workloadType: sts
17        fixedIP: "true"
18        app: fixedIPApp
19    spec:
20      containers:
21      - image: nginx
22        name: nginx

3. Verify IP allocation results

1# kubectl get pod {podName} -oyaml
2apiVersion: v1
3kind: Pod
4metadata:
5  annotations:
6    cce.baidubce.com/PodSubnetTopologySpread: example-subnet-topology
7  labels:
8    app: foo
9  name: foo-0
10  namespace: default
11spec:
12  affinity:
13    nodeAffinity:
14      requiredDuringSchedulingIgnoredDuringExecution:
15        nodeSelectorTerms:
16        - matchExpressions:
17          - key: topology.kubernetes.io/zone
18            operator: In
19            values:
20            - zoneF
21  containers:
22  - image: nginx
23    imagePullPolicy: IfNotPresent
24    name: nginx
25    resources:
26      limits:
27        cce.baidubce.com/ip: "1"
28      requests:
29        cce.baidubce.com/ip: "1"

Scenario 3: Fixed IP

Fixed IP policy means providing an IP list from which CCE assigns IP to stateful workloads pods. After Pod deletion, the IP address is retained. The IP address remains unchanged after Pod reconstruction or migration. Regardless of whether the pod is deleted, the allocated IP address will not be released and only be reclaimed until the workload is deleted.

Applicable scenarios:

• The specified subnet must be exclusively used by a single K8S Cluster CCE and will no longer support dynamic IP allocation. CCE will mark this subnet as a dedicated one.
• Implement security policies at the IP address level, such as configuring ACLs at the subnet level.
• Only applicable to pods created by stateful workloads (apps/v1 StatefulSet).

1. Create psts

1apiVersion: cce.baidubce.com/v2
2kind: PodSubnetTopologySpread
3metadata:
4  name: example-subnet-topology
5  namespace: default
6spec:
7  subnets:
8    sbn-6mrkdcsyzpaw:
9# Optional. Fixed IP range; if left blank, it indicates that the default IP range of the subnet is used
10    - family: 4
11      range:
12      - start: 10.0.0.2
13        end: 10.0.0.254
14  strategy:
15    releaseStrategy: Never
16    type: Fixed
17    enableReuseIPAddress: true
18# Select the pods to apply this subnet topology spread
19  selector:
20    matchLabels:
21      app: foo

2. Create workloads

1apiVersion: apps/v1
2# Stateful workloads must be used.
3kind: StatefulSet
4metadata:
5  name: foo
6  namespace: default
7spec:
8  replicas: 1
9  selector:
10    matchLabels:
11      app: fixedIPApp
12  serviceName: foo
13  template:
14    metadata:
15      labels:
16        workloadType: sts
17        fixedIP: "true"
18        app: fixedIPApp
19    spec:
20      containers:
21      - image: nginx
22        name: nginx

3. Verify IP allocation results

1# kubectl get pod {podName} -oyaml
2apiVersion: v1
3kind: Pod
4metadata:
5  annotations:
6    cce.baidubce.com/PodSubnetTopologySpread: example-subnet-topology
7  labels:
8    app: foo
9  name: foo-0
10  namespace: default
11spec:
12  affinity:
13    nodeAffinity:
14      requiredDuringSchedulingIgnoredDuringExecution:
15        nodeSelectorTerms:
16        - matchExpressions:
17          - key: topology.kubernetes.io/zone
18            operator: In
19            values:
20            - zoneF
21  containers:
22  - image: nginx
23    imagePullPolicy: IfNotPresent
24    name: nginx
25    resources:
26      limits:
27        cce.baidubce.com/ip: "1"
28      requests:
29        cce.baidubce.com/ip: "1"

Related products

• Virtual Private Cloud (VPC)