Cluster Pod Subnet Topology Distribution (Container Network v2)

Overview

The VPC-ENI container network mode supports assigning subnet to allocate IP for pods. Users can use this capability to plan and allocate IP addresses from different subnets for different business pods. This document introduces an extension of psts, namely the cluster-level subnet topology spread policy (cpsts).

Note: To enable this feature, please contact Baidu AI Cloud customer service to activate the allow list.

Requirement scenarios

psts provides three IP allocation policies to address scenarios involving dynamic IP address allocation, manual IP address allocation, and fixed IP. However, a single psts only takes effect within one namespace. To implement identical IP address allocation policies across different applications, multiple psts should be created, making reuse complicated. cpsts provides cluster-level subnet topology spread policies. It is only required to create a single cpsts object and it can take effect in multiple namespaces, resolving IP address allocation policy issues for multiple namespaces.

Solution overview

As demonstrated in the CRD data structure below, CCE provides a custom CRD to specify pod subnet topology spread policies for implementing subnet IP allocation in the K8S Cluster CCE.

Introduction to key data structure

Cluster Pod Subnet Topology Spread (ClusterPodSubnetTopologySpread)

The cluster subnet topology spread object (cpsts) will create pod subnet topology spread (psts) objects in eligible namespace according to namespaceSelector during working. It also affects the IP address allocation for pods based on psts. Its data structure is defined as follows:

1apiVersion: cce.baidubce.com/v2alpha1
2kind: ClusterPodSubnetTopologySpread
3metadata:
4  name: cpsts-1
5spec:
6  name: cpsts-1
7# Select the namespace where the cpsts object takes effect.
8  namespaceSelector:
9    matchLabels:
10      zrq: test
11# Select the pods where the cpsts object takes effect under each namespace
12  selector:
13    matchLabels:
14      app: redis
15  subnets:
16# When an IP range within the subnet is not specified, the subnet format is as follows (note the brackets)
17    # sbn-6mrkdcsyzpaw: []
18    sbn-6mrkdcsyzpaw:
19# Optional. Specify an IP range within this subnet. If left blank, the subnet cidr IP range will be used.
20      - family: 4
21        range:
22        - start: 10.0.0.2
23          end: 10.0.0.254
24  strategy:
25    releaseStrategy: TTL
26    ttl: 168h0m0s
27    type: Elastic
28# Enable IP reuse
29    enableReuseIPAddress: true
30  whenUnsatisfiable: DoNotSchedule

The core fields of this object are as follows:

Usage restrictions

1. This function requires the ENI cross-subnet IP allocation capability of VPC. Please submit a ticket to request activation for the ENI cross-subnet IP allocation feature.
2. Pods in the kube-system namespace cannot use the designated subnet IP allocation function.
3. When using ipRange capability, ensure the IP range excludes special IP address (e.g., IPv4 network address, gateway address, broadcast address, multicast address); otherwise, normal IP allocation may not be done.
4. Pods in specified subnets can only be scheduled to nodes located in the same availability zone as the subnet. Please ensure that the AZ contains nodes in Ready status.
5. In scenarios involving fixed IPs and IP reuse, only dedicated subnets (subnets exclusively reserved for a single K8S Cluster CCE) can be used. Dedicated subnets cannot be converted to general-purpose subnets or removed from the cluster. For details, please refer to the description of dedicated subnets.
6. This function is applicable only to clusters using container network v2.

Dedicated subnet: When the user needs to assign several IPs under the specified subnet to pods, the subnet of the IP will automatically be marked as manual IP allocation mode. Subnets in manual IP allocation mode have the following characteristics:

1. Dedicated subnets must be exclusive to the current K8S Cluster CCE. CCE automatically adds exclusive tags to subnets to prevent other K8S Cluster CCEs from using them. (However, users can operate other Baidu AI Cloud products to utilize such subnets.)
2. Dedicated subnets only support manual IP allocation and do not allow automatic allocation. Users must manage IP address planning and assignments themselves.
3. The relationship between IPs and pods in dedicated subnets supports two modes: priority allocation and fixed binding. The fixed binding policy uses the pod's name as the identifier, ensuring that pods with the same name always have the same IP address.
4. The default subnet for cluster pod cannot be a dedicated subnet; otherwise, other pods in the cluster may fail to allocate IP normally
5. It does not support converting dedicated subnets to regular subnets or deleting them from the cluster

Configuration steps

Environment preparation

Create private subnet

In the Baidu AI Cloud VPC Console -> Subnet tab, create a new subnet for your VPC and save the subnet ID (subnet IDs are named in the format sbn-xxx). Note: When creating a subnet, select an availability zone associated with the K8S Cluster CCE nodes; otherwise, scheduling may fail.

Description: To enable ENI cross-subnet IP allocation function, please submit a ticket request.

Allocate IP for specified subnets in CCE

Scenario 1: Cluster-wide multi-application fixed IP

Dynamically assign IP to pods from the subnet associated with the subnet topology constraint policy matching with pod. All pods matching the subnet topology constraint policy (cpsts) will only be assigned IP addresses from subnets included in the policy. Applicable scenarios:

• Perform traffic statistics by subnet dimension
• Implement security policies at the subnet dimension, such as ACL rule control
• Enable Internet access for specific subnets via NAT (Network Address Translation)

1. Create cpsts

1apiVersion: cce.baidubce.com/v2alpha1
2kind: ClusterPodSubnetTopologySpread
3metadata:
4  name: cpsts-1
5spec:
6  name: cpsts-1
7# Select the namespace where the cpsts object takes effect
8  namespaceSelector:
9    matchLabels:
10      zrq: test
11# Select the pods where the cpsts object takes effect under each namespace
12  selector:
13    matchLabels:
14      app: fixedIPApp
15  subnets:
16    sbn-6mrkdcsyzpaw:
17# Optional. Fixed IP range; if left blank, it indicates that the default IP range of the subnet is used
18      - family: 4
19        range:
20        - start: 10.0.0.2
21          end: 10.0.0.254
22  strategy:
23    releaseStrategy: TTL
24    ttl: 168h0m0s
25    type: Elastic
26# Enable IP reuse
27    enableReuseIPAddress: true
28  whenUnsatisfiable: DoNotSchedule

2. Create workloads

1apiVersion: apps/v1
2# Stateful workloads must be used
3kind: StatefulSet
4metadata:
5  name: foo
6  namespace: default
7spec:
8  replicas: 1
9  selector:
10    matchLabels:
11      app: fixedIPApp
12  serviceName: foo
13  template:
14    metadata:
15      labels:
16        workloadType: sts
17        fixedIP: "true"
18        app: fixedIPApp
19    spec:
20      containers:
21          - image: nginx
22            name: nginx

3. Verify IP allocation results

1# kubectl get pod {podName} -oyaml
2apiVersion: v1
3kind: Pod
4metadata:
5  annotations:
6    cce.baidubce.com/PodSubnetTopologySpread: example-subnet-topology
7  labels:
8    app: foo
9  name: foo-0
10  namespace: default
11spec:
12  affinity:
13    nodeAffinity:
14      requiredDuringSchedulingIgnoredDuringExecution:
15        nodeSelectorTerms:
16        - matchExpressions:
17          - key: topology.kubernetes.io/zone
18            operator: In
19            values:
20            - zoneF
21  containers:
22  - image: nginx
23    imagePullPolicy: IfNotPresent
24    name: nginx
25    resources:
26      limits:
27        cce.baidubce.com/ip: "1"
28      requests:
29        cce.baidubce.com/ip: "1"

Related products

• Virtual Private Cloud (VPC)