CCE Network Plugin Description

Component introduction

The CCE Network Plugin is a CNI plugin developed based on Baidu AI Cloud Elastic Network Interface.

Component function

The CCE Network Plugin mode is an extended networking mode supported by Cloud Container Engine (CCE). Utilizing Baidu AI Cloud's elastic network interface offerings, it assigns IP addresses within the VPC to Pods in the cluster. Baidu AI Cloud's VPC functionality ensures routing and connectivity within the container network, enabling the control plane and data plane of pods and nodes to operate on the same network layer. In this mode, pods can leverage all product features of Baidu AI Cloud VPC.

Usage restrictions

• CCE Network Plugin is a system component. Currently, it does not support upgrades through the console. If you need an upgrade, please submit a ticket

Install component

Install component As a system component, CCE Network Plugin is installed by default when creating a cluster, so you do not need to pay attention to the installation here.

Deployment status

After installing the CCE Network Plugin component, the following objects will be deployed in the Kubernetes cluster:

Object name	Namespace	Resource type	Resource usage
cceendpoints.cce.baidubce.com	-	CRD	-
clusterpodsubnettopologyspreads.cce.baidubce.com	-	CRD	-
enis.cce.baidubce.com	-	CRD	-
netresourcesets.cce.baidubce.com	-	CRD	-
podsubnettopologyspreads.cce.baidubce.com	-	CRD	-
subnets.cce.baidubce.com	-	CRD	-
cce-network-v2-reserved	kube-system	-	ConfigMap
cce-network-v2-config	kube-system	-	ConfigMap
cni-config-template	kube-system	-	ConfigMap
cce-network-operator	kube-system	-	Deployment
cce-network-agent	kube-system	-	DaemonSet
cce-network-v2	kube-system	-	Deployment
cce-cni-v2	kube-system	-	ServiceAccount
cce-network-v2	kube-system	-	Deployment
cce-cni-v2	kube-system	-	ClusterRole
cce-cni-v2	kube-system	-	ClusterRoleBinding
cce-network-v2-mutating-webhook	kube-system	-	MutatingWebhookConfiguration

Major version changes

Version No.	Cluster version compatibility	Update time	Update content	Limitations and impacts
v2.12	CCE/v1.18+	2024.06.28	New feature: Support Burstable ENI pool, effectively avoiding insufficient ENI pool resources on nodes when subnet IP resources are tight. New feature: Add ENI security group synchronization function to keep CCE ENI and node security groups in sync. New feature: Add node network configuration set function NetResourceConfigSet, and support specified nodes to configure network resources independently.	None
v2.11	CCE/v1.18+	2024.05.27	New Feature: Adds support for allocating RDMA subnet interface cards and RDMA secondary IP addresses within containers.	None
v2.10	CCE/v1.18+	2024.03.05	New feature: Support the secondary IP address mode of EBC primary network interface card in VPC-ENI mode. New feature: Refractory CNI configuration file management logic to support retaining custom CNI plugin configurations. New feature: Add support for the portmap plugin, enabled by default. New feature: VPC-ENI supports automatic acquisition of node ENI quota information, removing parameters for custom ENI quotas. New feature: Support specifying the maximum number of secondary IP addresses for ENIs on a node by adding the annotation network.cce.baidubce.com/node-eni-max-ips-num to the node.	None
v2.9	CCE/v1.18+	2023.11.10	New CRD: Support cluster-level psts ClusterPodSubnetTopologyStrategy (cpsts). A single cpsts can control psts policies applied to the entire cluster. CRD field changes: The NetworkResourceSet resource pool has added abnormal statuses of ENIs on nodes, reporting single-machine IP capacity status and overall ENI network interface card status. New features: Support Ubuntu 22.04 OS. In the container network environment, the MacAddressPolicy of systemd-networkd is defined as none. New feature: Support pod-level QoS.	None

Detailed version change records

2.12

New feature functions:

1. Supports the Burstable ENI pool, effectively preventing resource shortages in the ENI pool on nodes when subnet IP resources are constrained.
2. Introduces an ENI security group synchronization feature to ensure CCE ENIs and node security groups remain aligned.
3. Introduces the NetResourceConfigSet function, allowing specific nodes to independently configure network resources.

2.12.1 [2024/07/02]

1. [Bug] Fix the null pointer issue during initialization when burstable ENI is enabled for BBC models
2. [Bug] Fix the issue where ENI cannot be selected when BBC ENI does not return an instance ID, affecting node readiness time

2.12.0 [2024/06/28]

1. [Feature] Support Burstable ENI Pool, effectively avoiding insufficient ENI pool resources on nodes when subnet IP resources are tight.
2. [Feature] Add metrics for ENI creation failures caused by insufficient subnets
3. [Feature] Add ENI security group synchronization function to keep CCE ENI and node security groups in sync
4. [Feature] Optimize Pod scheduling algorithm, and add automatic adaptation of node IP capacity to avoid waste of node IP address resources
5. [Feature] Add node network configuration set function NetResourceConfigSet, and support specified nodes to configure network resources independently
6. [Optimize] Fix the issue where the psts object may update cep with a null addressing when using enableReuseIPAddress, preventing the recording of error messages
7. [Optimize] Optimize the operator event backlog issue to avoid long-term timeout backlogs of events
8. [Optimize] Use agent to optimize the IP address gc algorithm and support the ability to clean up changed cep legacy addresses based on IP addresses after reaching the gc cycle
9. [Optimize] Bound dynamic cep with the lifecycle of nrs to reduce the number of legacy cep objects when the agent is killed during scale-down
10. [Optimize] Optimize the rdma IP application process to avoid cep using fixed IP for rdma

2.11 (2024/5/27)

New feature functions:

1. New Feature: Adds support for allocating RDMA subnet interface cards and RDMA secondary IP addresses within containers.

2.11.3 [Unreleased]

1. [Feature] --endpoint-gc-intervalAdd the minimum interval time for control agents to update nrs
2. [Optimize] Optimize the processing logic for eni restart events and agent restart speed
3. [Optimize] Reorder IP addresses of bce eni to reduce unnecessary ENI update events
4. [Bug] Optimize the possible null pointer in StartSynchronizingSubnet when subnets are manually deleted

2.11.2 [20240616]

1. [Bug] Fix the issue of continuous error retries after deleting nrs and the continuously creating eni
2. [Bug] Fix the restore failure issue after agent restart
3. [Feature] Add ehc model support to VPC-ENI
4. [Optimize] Add alloc-worker to cce-network-operator to allow configuration of the number of coroutines for concurrent processing of nrs objects
5. [Optimize] Optimize rdma to pre-apply for 13 IPs by default, with a maximum of 104 idle IPs, to avoid frequent IP applications and releases.
6. [Bug] Fix the possible null pointer issue when releasing rdma IPs
7. [Optimize] Optimize the logic for creating new subnets for ebc hosts. In the secondary IP address mode of a non-primary network interface card, the primary network interface card subnet is no longer validated
8. [Optimize] Remove redundant operator logs to reduce resource usage by the operator
9. [Optimize] Update the latest subnet information when restarting the agent to avoid inconsistent subnet information after restart
10. [Optimize] Reduce the scope of unnecessary locks in the operator to improve operator processing performance
11. [Optimize] Add a forced end time for triggers to avoid a single node being stuck and affecting overall synchronization
12. [Optimize] Add HPC eni OpenAPI interface rate limiting
13. [Optimize] Merge rdma and Ethernet resource synchronizers to reduce resource overhead from repeated synchronization
14. [Feature] Add an IP release and reclaim control switch; ENI IPs are not reclaimed by default
15. [Bug] Fix the occasional issue where the maintainIPPool method is not called on nodes, preventing node synchronization
16. [Bug] Fix the concurrent access issue of the bcesync map
17. [Optimize] Add corresponding request ID to events when subnet IPs are insufficient
18. [Feature] Add the granularity of trigger metrics, refined to nodes

2.11.1 [20240611]

1. [Optimize] Update the configuration method for RDMA IPPool MinAllocateIPs/PreAllocate/MaxAboveWatermark parameters to be consistent with VPC-ENI
2. [Optimize] Retain the original name of RDMA network interface cards and no longer rename RDMA interface cards to avoid loss of RDMA-related policy routes on nodes
3. [Bug] Fix RDMA Discovery startup failure due to missing ENISpec in VPC routes
4. [Bug] Fix the unsynchronized status issue of RDMA network interface cards in ENI objects
5. [Bug] Fix incorrect calculation of the maximum number of IPs for RDMA network interface cards and optimize error messages
6. [Bug] Fix the misjudgment of the roce plugin as a custom user plugin

2.11.0 [20240527]

1. [Feature] New feature: Support allocation of RDMA network interface cards within containers a. Support single-container allocation of RDMA network interface cards in addition to VPC Ethernet network interface cards, including ERI and eRDMA interface cards. b. Containers use RDMA network interface cards in shared mode. All containers using RDMA resources on a single node share RDMA network interface card. Each RDMA network interface card creates sub-devices with independent RDMA IPs within containers.

2.10 (2024/03/05)

New feature functions:

1. New feature: Added support for the secondary IP address mode of the EBC primary network interface card in VPC-ENI mode.
2. New feature: Refined CNI configuration file management logic to retain custom CNI plugin configurations.
3. New feature: Enabled support for the portmap plugin by default.
4. New feature: VPC-ENI now supports automatic acquisition of node ENI quota information, eliminating the need for custom ENI quota parameters.
5. New feature: Support specifying the maximum number of secondary IP addresses for ENIs on a node by adding the annotation network.cce.baidubce.com/node-eni-max-ips-num to the node.

2.10.4/2.10.5 [202405011]

1. [Bug] Fix incorrect calculation of the maximum IP address capacity per machine in VPC-ENI mode

2.10.3 [20240425]

1. [Bug] Fix the issue where informers repeatedly add processors when ResyncController has already added EventHandler, causing psts to receive duplicate events and leading to IP address conflicts

2.10.2 [20240403]

1. [Bug] Fix the cni file rewriting error in vpc-route mode

2.10.1 [20240325]

1. [Bug] Fix the issue where restarting the operator in vpc-route mode might cause cidr duplication across multiple nodes
2. [Bug] Fix the stack overflow issue that may occur when calling the bce sdk, causing operator restarts
3. [Optimize] Add mac address validity check for vpc-eni to avoid misoperation on other network interface cards

2.10.0 （2024/03/05）

1. [Feature] VPC-ENI supports automatic acquisition of node eni quota information, removing parameters for custom ENI quotas.
2. [Feature] VPC-ENI BBC supports the secondary IP address mode of the primary network interface card
3. [Feature] VPC-ENI BBC upgrades the secondary IP address mode of the primary network interface card
4. [Optimize] Add CNI plugin log persistence
5. [Feature] Refractory CNI configuration file management logic to support retaining custom CNI plugin configurations
6. [Feature] Add support for the portmap plugin, enabled by default
7. [Feature] Support specifying the maximum number of secondary IP addresses for ENIs on a node by adding the annotation network.cce.baidubce.com/node-eni-max-ips-num to the node.
8. [Bug] Fix the issue where cni plugins cannot execute on arm64 architecture
9. [Optimize] Add BCE SDK log persistence
10. [Optimize] Optimize the backoff and retry strategy for removing bce sdk to avoid frequent retries
11. [Optimize] Support using the default-api-timeout custom parameter to specify the BCE OpenAPI timeout duration

2.9 (2023/11/10)

New feature functions:

1. New CRD: Introduced support for the cluster-level ClusterPodSubnetTopologyStrategy (cpsts). A single cpsts can now manage psts policies across the entire cluster.
2. CRD field changes: The NetworkResourceSet resource pool now includes abnormal statuses of ENIs on nodes, reports on single-machine IP capacity, and overall ENI network interface card statuses.
3. New features: Added support for Ubuntu 22.04 OS. In container network environments, the MacAddressPolicy of systemd-networkd is now defined as "none."
4. New feature: Support pod-level QoS.

2.9.5 [20240325]

1. [Bug] Fix the issue where restarting the operator in vpc-route mode might cause cidr duplication across multiple nodes
2. [Bug] Fix the stack overflow issue that may occur when calling the bce sdk, causing operator restarts

2.9.4 [20240305]

1. [Feature] Support BBC instances to configure the subnet for ENIs on a node by adding the network.cce.baidubce.com/node-eni-subnet anotation on the node.

2.9.3 [20240228]

1. [Feature] cce-network-agent automatically synchronizes node annotation information to CRD.
2. [Feature] Support EBC instances to configure the subnet for ENIs on a node by adding the network.cce.baidubce.com/node-eni-subnet anotation on the node.
3. [Feature] Add the enable-node-annotation-sync parameter, disabled by default.
4. [Bug] Correct the calculation error of the number of ENIs that can be created when pre-applying IPs.

2.9.2 [20240223]

1. [Bug] Fix the issue where cni plugins cannot execute on arm64 architecture

2.9.1 [20240115]

1. [Optimize] Optimize the lock handling of NetResourceManager when receiving events, eliminating 6-minute delays in event processing
2. [Optimize] Add 3 retry attempts when ENI state machine synchronization fails, eliminating 10-minute readiness delays caused by ENI state delays
3. [Bug] Fix the issue where cce-network-agent incorrectly identifies OS information
4. [Bug] Fix the issue where the operator may exit with a null pointer after the cce-network-agent pod is deleted
5. [Bug] Fix the issue where events cannot be printed on nrs objects when creating eni

2.9.0 [20240102]

1. [Optimize] When IP application fails, support providing failure reasons, including: a. No available subnets b. IP address pool is full c. Node ENI pool is full d. No available IPs in the subnet e. IP cache pool exceeds limit
2. [Feature] Add CRD: ClusterPodSubnetTopologyStrategy (cpsts), used to control cluster-level psts policies. a. Current crd version: cce.baidu.com/v2beta1 b. cpsts supports configuring psts policies for all namespaces matching namespaceSelector and manages their lifecycle and status as child objects.
3. [Feature] Support Ubuntu 22.04 OS. In the container network environment, the MacAddressPolicy of systemd-networkd is defined as none.
4. [Feature] Support Pod-level bandwidth control and control Pod-level bandwidth by setting annotations on Pods. a. kubernetes.io/ingress-bandwidth: 10 M Configure the Pod’s ingress bandwidth to 10 M b. kubernetes.io/egress-bandwidth: 10 M Configure the Pod’s egress bandwidth to 10 M
5. [Feature] Support Pod-level QoS and control the QoS of Pods by setting annotations on Pods. a. cce.baidubce.com/egress-priority: Guaranteed Configure the Pod’s traffic to Guaranteed (lowest latency) priority b. cce.baidubce.com/egress-priority: Burstable Configure the Pod’s traffic to Burstable (high priority) c. cce.baidubce.com/egress-priority: BestEffort Configure the Pod’s egress traffic to low priority
6. [Optimize] Modify the logic of --bce-customer-max-eni and --bce-customer-max-ip parameters. When the parameters are non-zero, they take effect forcefully
7. [Bug] Fix the issue where netns cannot be read when the container network namespace mount type is tmpfs in exclusive eni mode
8. [Feature] Add the override-cni-config switch, and forcibly overwrite the cni configuration file by default when the agent starts
9. [Feature] Add affinity scheduling function when psts reuses IPs to ensure that Pods with the same name can be scheduled to the same availability zone to reuse subnets during repeated scheduling.
10. [Optimize] Optimize the logic for concurrent ENI creation to avoid creating excessive ENIs concurrently when the service does not require many IPs
11. [Optimize] Optimize ENI naming length, limited to 64 characters
12. [Bug] Fix the issue where Pods may apply for expired IP addresses during concurrent IP application and release in VPC-ENI mode

2.8 (2023/08/07)

1. Official release of container network v2

2.8.8 [20231227]

1. [Bug] Pods may apply for expired IP addresses during concurrent IP application and release in VPC-ENI mode

2.8.7 [20231127]

1. [Bug] Fix the ineffective configuration of --bce-customer-max-eni and --bce-customer-max-ip parameters in cce-network-v2-config; no limit on concurrent ENI creation, which may lead to exceeding the maximum number of ENIs under concurrency

2.8.6 [20231110]

1. [Bug] Optimize the logic where EndpointManager does not time out when updating endpoint objects and may enter an infinite loop due to resource expiration
2. [Optimize] Optimize the operator work queue to support custom worker counts and accelerate event processing
3. [Optimize] Change core workflow logs of EndpointManager to info level
4. [Optimize] Optimize the EndpointManager gc workflow, and set the gc time for dynamic IP allocation to one week
5. [Optimize] Add re-enqueue time when ENI VPC state machine transitions do not trigger state changes to accelerate ENI readiness time
6. [Optimize] Add ENI addition/deletion state change events and log record for non-terminal statuses of ENI VPC
7. [Optimize] Record relevant events when metaapi is missing
8. [Optimize] Record relevant events when VPC routes are full

2.8.5 [20241017]

1. [Optimize] Optimize the reclaim mechanism for failed IP allocation in psts to avoid IP leakage
2. [Bug] Fix the issue where nrs finalizers cannot be reclaimed because the vpc route state is released after nrs is marked with deleteTimeStamp in vpc route mode
3. [Optimize] Optimize the logic for creating cep. When cep creation fails, attempt to actively delete and recreate cep

2.8.4 [20230914]

1. [Bug] For vpc-eni, fix the issue where ENI cannot be ready due to DHCP deleting IPs after ENI network interface cards are renamed in OS using NetworkManager such as CentOS 8

2.8.3 [20230904]

1. [Feature] Support re-creation of cni configuration files after they are deleted by kubelet
2. [Feature] network-agent supports enabling pprof and acquiring mutex and block data
3. [Optimize] Remove the fill lock when network-agent applies for IPs
4. [Bug] Fix the default rate limiting configuration of network-agent

2.8.2 [20230829]

1. [Optimize] Improve ENI creation performance and shorten nrs task management time
2. [Optimize] Add logic for concurrent pre-creation of ENI. When the number of preloaded ENIs per machine is not reached, create ENIs concurrently
3. [Bug] Fix the issue where querying ENI objects by ENI name returns null during ENI creation, resulting in a minimum 1-minute creation time per ENI