LoadBalancer Service Annotation Description

LoadBalancer Service Annotation Description

When utilizing LoadBalancer-type service resources (LB Service) provided by CCE, advanced configurations can be applied by adding annotations.

The annotation is located at the following position in the complete service YAML:

1apiVersion: v1
2kind: Service
3metadata:
4  name: service-example
5  annotations:
6    prometheus.io/scrape: "true"
7 service.beta.kubernetes.io/cce-load-balancer-internal-vpc: "true" # LB Service does not assign an EIP
8……

LoadBalancer Service can use the following advanced configuration items:

Specify the application BLB to use

Value description

If set, CCE will configure LB Service rules on the specified application BLB;

If this annotation is absent, CCE will create a new application BLB and automatically assign the annotation to the new application's BLB ID.

Notes

• Only application BLB is supported
• You must first create an application BLB, which should reside in the same VPC as the cluster
• Modifying or deleting certain settings may expose the BLB. Be cautious not to overwrite or omit critical annotations when altering LB Service objects.

Example

service.beta.kubernetes.io/cce-load-balancer-id: lb-12345678

LB Service selects to mount target nodes according to labels.

Value description

The service can use the annotation service.beta.kubernetes.io/cce-load-balancer-backend-label: "k1=v1, k2=v2 ..." to select nodes that match the label for binding

If no label is added, the service backend will be drained, and an event will be recorded in the service description.

Notes

• Multiple labels are separated by commas, e.g., "k1=v1,k2=v2", with an AND relationship between them;
• When the selector defined in the annotation service.beta.kubernetes.io/cce-load-balancer-backend-label: "k1=v1, k2=v2 ..." matches zero nodes, the service backend will be drained, resulting in a service outage. When using this function, certain management of the cluster node labels is required;
• Adding compliant nodes or modifying existing nodes will also trigger updates;
• Service local mode inherently has node-selection capabilities. For this reason, avoid explicitly specifying the access layer backend while in service local mode.

Example

1apiVersion: v1
2kind: Service
3metadata:
4  name: service-example
5  annotations:
6 service.beta.kubernetes.io/cce-load-balancer-backend-label: k1=v1, k2=v2 #  There is an AND relationship between multiple labels
7......

Specify EIP address/Retain EIP resources upon deletion

Value description

This configuration must be set under the Spec field, not within Annotations.

By configuring an EIP address in spec.loadBalancerIP, CCE will bind the EIP to the application BLB corresponding to LB Service; and the EIP resource will not be deleted when the LB Service resource is deleted;

If this setting is omitted, CCE will automatically create a new EIP by default, leaving this field blank.

Notes

• Prepaid EIP resources can be specified;
• Modification or deletion may lead to EIP exposure. Do not overwrite or lose this configuration when modifying LB Service objects;
• If it is not added during creation, it can be added before deletion, and adding it before deletion remains effective;
• If an EIP is already associated with the LB Service, modifying this annotation will not replace the existing EIP address linked to the application BLB. Users must rebind it manually.

Example

1kind: Service
2apiVersion: v1
3metadata:
4    name: test
5spec:
6    type: LoadBalancer
7    loadBalancerIP: 8.8.8.8
8......

Set the BLB name associated with the LB Service

Value description

This annotation defines the BLB name associated with the LB Service; otherwise, the default naming structure (cluster ID/svc/namespace/service name) will be used to create the BLB.

Occasionally, the default name may exceed the maximum limit of 64 characters. In such instances, manually set the value of this annotation to specify the BLB name.

Notes

• Do not assign a name identical to an existing BLB to prevent mismanagement or improper use of the BLB.

Example

service.beta.kubernetes.io/cce-load-balancer-lb-name: cce-test-name

Retain application BLB resources upon deletion

Value description

If the value is true, the associated application BLB resource will not be deleted when the LB Service object is deleted;

If missing or the value is false, the associated application BLB will be deleted when the LB Service object is deleted;

Notes

• If this annotation is not added at creation, it can be added before deletion, and doing so will still be effective.

Example

service.beta.kubernetes.io/cce-load-balancer-reserve-lb: "true"

Specify the subnet for the newly created BLB

Value description

If the value is a valid subnet ID of the cluster's VPC, CCE will create the application BLB in this subnet when a new application BLB is needed;

In the absence of this annotation, CCE will independently choose a subnet to create an application BLB.

Notes

• This setting is applicable only to newly created application BLBs.

Example

service.beta.kubernetes.io/cce-load-balancer-subnet-id: "sbn-12345678"

Configure LB Service for VPC intranet access only

Value description

If the value is true, CCE will not create or set an EIP address for it;

If missing or the value is false, CCE will allocate and bind an EIP to the application BLB.

Notes

• If an EIP is already associated with the application BLB, adding this annotation will not automatically unbind it in the backend, so users must handle the unbinding manually.

Example

service.beta.kubernetes.io/cce-load-balancer-internal-vpc: "true"

Custom EIP configuration

Value description

If the LB Service does not specify an EIP and is not configured exclusively for VPC intranet access, new EIP resources will be created automatically.

The configuration for newly created EIPs can be set in the annotation:

• Specify the charge type:

Support ByTraffic (pay-as-you-go) and ByBandwidth (pay-by-bandwidth).

The corresponding annotation is service.beta.kubernetes.io/cce-elastic-ip-billing-method.

• Specify the route type:

Support two route types: Enhanced BGP and standard BGP. CCE defaults to support the corresponding annotation service.beta.kubernetes.io/cce-elastic-ip-route-type, where BGP_S represents enhanced BGP, BGP represents standard BGP, ChinaTelcom represents China Telecom single-line, ChinaUnicom represents China Unicom single-line, and ChinaMobile represents China Mobile single-line. Usage example is as follows:

Example

1# Enhance BGP
2service.beta.kubernetes.io/cce-elastic-ip-route-type:'BGP_S'

• Specify bandwidth:

The valid range is [1, 5000].

The corresponding annotation is service.beta.kubernetes.io/cce-elastic-ip-bandwidth-in-mbps.

Notes

• New EIP is postpay type by default; using annotations to specify prepay EIP creation is currently unsupported;
• Billing method and route type are only valid for newly created EIPs. EIP bandwidth can be modified, but the full configuration update (real-time configuration update) annotation service.kubernetes.io/cce-update-in-time: "true" must be added for changes to take effect.
• When the bandwidth quota is not a valid number, a warning event will be triggered;
• When you do not add a bandwidth quota annotation, CCE will create and bind a standard EIP with a default bandwidth quota of 100 and a pay-as-you-go type;
• Enhanced BGP creation is temporarily unavailable in some overseas regions. For details, please refer to;
• The valid range for bandwidth quota varies depending on the combination of internet connection type and bill type, as shown in the table below:

• Failure to meet any of the following four conditions will result in EIP creation failure:

Hide the intranet IP address of the application BLB

Value description

If the value is true, the intranet IP field of the application BLB will no longer be displayed in the service's status;

If missing or the value is false, the intranet IP field of the application BLB will be normally displayed in the service's status.

Example

service.beta.kubernetes.io/cce-load-balancer-external: "true"

Set the maximum number of backends mounted by default for BLB

Value description

This annotation helps limit the number of backends mounted by the Load Balancer (LB) Service.

The default value is set to 200.

Notes

• If the service's Spec.ExternalTrafficPolicy is configured as "local" or Pod direct connection mode is enabled, make sure this value exceeds the number of cluster nodes. Otherwise, nodes or Pods not linked to the BLB backend will not receive traffic.

Example

service.beta.kubernetes.io/cce-load-balancer-rs-max-num: 500

Use pod direct connection mode

Value description

When adding the annotation service.beta.kubernetes.io/cce-load-balancer-backend-type: "eni", the application BLB will be configured with Pod direct connection mode, that is, the traffic will directly reach the backend Pod from the application BLB.

Notes

• Avoid adding this annotation to existing ones to switch to Pod direct connection mode. If switching is needed, create a new LB Service instead, and vice versa.

Example

service.beta.kubernetes.io/cce-load-balancer-backend-type: "eni"

Full configuration update (real-time configuration updates)

Value description

• CCE does not automatically synchronize all BLB configurations. Even if updates are made, configurations such as BLB listener settings, health check rules, EIP bandwidth setups, or BLB security group rules will not be actively synchronized by CCE.
• If this annotation is set to true, CCE will fully update the BLB configuration: after modifying BLB listener configurations, health check rules, EIP bandwidth configurations, or BLB security group rules, the BLB and EIP configurations will be updated immediately;

Notes

• CCE does not automatically add this annotation; it must be manually added as needed;

Example

service.kubernetes.io/cce-update-in-time: "true"

Configure BLB security group

Value description

The service can use the annotation service.beta.kubernetes.io/cce.security.groups: "sgID1, sgID2 ...” to define the security groups associated with BLB instances, supporting multiple security group IDs separated by commas.

Notes

• CCE does not automatically add this annotation; it must be manually added as needed. If this annotation is not set, CCE will not associate a security group with the BLB instance by default;
• It must be used together with the service.kubernetes.io/cce-update-in-time: "true” annotation to trigger security group configuration updates for the BLB instance

Example

service.beta.kubernetes.io/cce.security.groups: 'sgID1,sgID2'

Use dedicated BLB clusters

Value description

If a dedicated BLB cluster is available, the backend will prioritize creating the application BLB within it.

The following annotations can be used to specify whether to use a dedicated cluster and which dedicated cluster ID will be used:

• Create an application BLB using a shared cluster

After using this annotationservice.beta.kubernetes.io/no-use-blb-cluster-l4: "true", the application BLB will be created on a shared cluster;

• Specify creation on a dedicated L4 cluster

After using this annotationservice.beta.kubernetes.io/no-use-blb-cluster-l4: "test", the application BLB will be created on a specified dedicated BLB cluster.

Notes

• For dedicated clusters, if no specific annotation is added, the backend will automatically select the dedicated BLB cluster for creation.

Example

1service.beta.kubernetes.io/no-use-blb-cluster-l4: "true"

Extend service port protocols

Value description

Extension protocol configurations are used to enhance the protocol settings for ports specified in the spec, and these will apply to the application BLB's listeners.

The ports defined in the extension protocol must already exist in the Spec configuration; otherwise, the extension protocol configuration will not take effect.

Extension protocol details are specified in the service.beta.kubernetes.io/cce.listener.customized-listener annotation. Refer to examples for detailed configuration. Configurable parameters for each protocol include the following:

TCP extension protocol

UDP extension protocol

TCP-SSL extension protocol

HTTP extension protocol

HTTPS extension protocol

Notes

• CCE does not automatically add this annotation; it must be manually added as needed;
• Only when the port protocol in Spec is TCP, TCP, TCP-SSL, HTTP and HTTPS extension protocol configurations can be added for it. If the port is UDP protocol in Spec, only UDP extension protocol configuration can be added for it;
• Changing the extension protocol types or deleting the extension protocol configuration for a port will initiate listener reconstruction for the corresponding application BLB port. Use with caution.

Example

1service.beta.kubernetes.io/cce.listener.customized-listener: |-
2      {
3         "81":{
4            "protocol":"TCP",
5            "timeout":"31",
6            "scheduler":"Hash"
7         },
8         "82":{
9            "protocol":"UDP",
10            "timeout":"31",
11            "scheduler":"Hash"
12         },
13         "83":{
14            "protocol":"HTTP",
15            "redirectPort":"84",
16            "timeout":"31",
17            "scheduler":"LeastConnection",
18            "enableKeepSession":"true",
19            "keepSessionType":"rewrite",
20            "disableXForwardedForHeader": "true",
21            "enableXForwardedProtoHeader": "true"
22         },
23         "84":{
24            "protocol":"HTTPS",
25            "serverCertIDs":"cert-5bqpykggjzmi",
26            "timeout":"31",
27            "scheduler":"LeastConnection",
28            "enableKeepSession":"true",
29            "keepSessionType":"rewrite",
30            "encryptionType": "userDefind",
31            "encryptionProtocols": "tlsv10, tlsv11, tlsv12",
32            "appliedCiphers": "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES128-SHA"
33         },
34         "85":{
35            "protocol":"TCP-SSL",
36            "serverCertIDs":"cert-5bqpykggjzmi",
37            "timeout":"31",
38            "scheduler":"LeastConnection",
39            "encryptionType": "userDefind",
40            "encryptionProtocols": "tlsv10, tlsv11, tlsv12",
41            "appliedCiphers": "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES128-SHA"
42         }
43      }

Load Balancer Service Example

1apiVersion: v1
2kind: Service
3metadata:
4  annotations:
5    service.beta.kubernetes.io/cce.listener.customized-listener: |-
6      {
7         "81":{
8            "protocol":"TCP",
9            "timeout":"31",
10            "scheduler":"Hash"
11         },
12         "82":{
13            "protocol":"UDP",
14            "timeout":"31",
15            "scheduler":"Hash"
16         },
17         "83":{
18            "protocol":"HTTP",
19            "redirectPort":"84",
20            "timeout":"31",
21            "scheduler":"LeastConnection",
22            "enableKeepSession":"true",
23            "keepSessionType":"rewrite",
24            "disableXForwardedForHeader": "true",
25            "enableXForwardedProtoHeader": "true"
26         },
27         "84":{
28            "protocol":"HTTPS",
29            "serverCertIDs":"cert-5bqpykggjzmi",
30            "timeout":"31",
31            "scheduler":"LeastConnection",
32            "enableKeepSession":"true",
33            "keepSessionType":"rewrite",
34            "encryptionType": "userDefind",
35            "encryptionProtocols": "tlsv10, tlsv11, tlsv12",
36            "appliedCiphers": "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-RSA-AES128-SHA"
37         },
38         "85":{
39            "protocol":"TCP-SSL",
40            "serverCertIDs":"cert-5bqpykggjzmi",
41            "timeout":"31",
42            "scheduler":"LeastConnection"
43         }
44      }
45  name: service-example
46  namespace: default
47spec:
48  clusterIP: 172.16.27.138
49  externalTrafficPolicy: Cluster
50  ports:
51    - name: nginx-1
52      nodePort: 30710
53      port: 81
54      protocol: TCP
55      targetPort: 80
56    - name: nginx-2
57      nodePort: 30792
58      port: 82
59      protocol: UDP
60      targetPort: 80
61    - name: nginx-3
62      nodePort: 31906
63      port: 83
64      protocol: TCP
65      targetPort: 80
66    - name: nginx-4
67      nodePort: 32514
68      port: 84
69      protocol: TCP
70      targetPort: 80
71    - name: nginx-5
72      nodePort: 31749
73      port: 85
74      protocol: TCP
75      targetPort: 80
76    - name: nginx-6
77      nodePort: 31569
78      port: 86
79      protocol: TCP
80      targetPort: 80
81  selector:
82    app: nginx
83  sessionAffinity: None
84  type: LoadBalancer

Configure server group dynamic weights

Value description

When this annotation is added, and the LoadBalancer Service's ExternalTrafficPolicy is set to local mode without Pod direct connection enabled, CCE will assign weights to the BLB's real servers based on the number of Pods on each node. Otherwise, CCE assigns equal weights to all backend nodes.

This helps prevent Pod load imbalance in local mode for ExternalTrafficPolicy.

Notes

• CCE does not add this annotation automatically; it must be added manually when necessary.
• If the total number of Pods is low and the average load is high, rolling updates may temporarily overload certain Pods. Avoid using this setting in such situations.

Example

1service.beta.kubernetes.io/cce-dynamic-backend-server-weight: 'true'

Set server group health check

Value description

If set, CCE will configure the corresponding health check for the BLB's server group;

If omitted, CCE will use the default health check configuration for the BLB's server group.

Notes

• Default health check configurations may cause failures; custom health check configurations are recommended. For more details, refer to Introduction to Baidu Load Balancer (BLB) Health Checks.

Example

1# Different ports of the LB Service may use different protocols, so health checks must be configured separately for each port
2 ## 1. Service listener port is 80, and health check protocol is TCP
3service.beta.kubernetes.io/cce.listener.customized-health-check:  |-
4      {
5         "80":{
6 "healthCheck":"TCP", #Health check protocol
7 "healthCheckTimeoutInSecond":"3", #Response timeout duration
8 "healthCheckIntervalInSecond":"3", #Health check interval
9 "healthCheckDownRetry":"3", #Unhealthy threshold
10 "healthCheckUpRetry": "3" #Health threshold
11         }
12      }
13      
14 ## 2. Service listener port is 80, and health check protocol is UDP
15 service.beta.kubernetes.io/cce.listener.customized-health-check:  |-
16      {
17         "80":{
18            "healthCheck":"UDP",
19 "udpHealthCheckString": "\00\01\01\00\00\01\00\00\00\00\00\00\05baidu\03com\00\00\01\00\01", #Check string
20            "healthCheckTimeoutInSecond":"3",
21            "healthCheckIntervalInSecond":"3",
22            "healthCheckDownRetry":"3",
23            "healthCheckUpRetry": "3"
24         }
25      }
26      
27 ## 3. Service listener port is 80, and health check protocol is ICMP
28 service.beta.kubernetes.io/cce.listener.customized-health-check:  |-
29      {
30         "80":{
31            "healthCheck":"ICMP",
32            "healthCheckTimeoutInSecond":"3",
33            "healthCheckIntervalInSecond":"3",
34            "healthCheckDownRetry":"3",
35            "healthCheckUpRetry": "3"
36         }
37      }
38      
39 ## 4. Service listener port is 80, and health check protocol is HTTP
40service.beta.kubernetes.io/cce.listener.customized-health-check:  |-
41      {
42         "80":{
43            "healthCheck":"HTTP",
44 "healthCheckHost": "baidu.com", #Host header
45 "healthCheckPort": "80", #Check port
46 "healthCheckUrlPath": "/", #Check path
47            "healthCheckTimeoutInSecond":"2",
48            "healthCheckIntervalInSecond":"2",
49            "healthCheckDownRetry":"3",
50            "healthCheckUpRetry": "2",
51 "healthCheckNormalStatus": "http_2xx|http_3xx" #Normal status codes, options include "http_2xx", "http_3xx", "http_4xx", "http_5xx", supporting multiple selections
52         }
53      }