CCE NodeLocal DNSCache Description
Component introduction
The CCE NodeLocal DNSCache component provides a DNS local cache solution based on the open-source NodeLocal DNSCache project. Deploying NodeLocal DNS in a CCE Kubernetes cluster enhances the stability and performance of service discovery. It improves cluster DNS performance by deploying a DNS cache proxy as a DaemonSet on cluster nodes. This component includes two main parts: NodeLocal DNSCache for local cache and the dynamic injection controller that handles DNSConfig injection deployment.
- The NodeLocal DNSCache creates a Pod on each node that listens to a local IP address (default IP: 169.254.20.10). Using the injected DNSConfig configuration, DNS requests from the Pod are proxied through the Local DNS local cache.
The Node Local DNS service is built on CoreDNS but focuses solely on providing proxy and cache functionality.
- The dynamic injection controller for DNSConfig uses the Admission Webhook mechanism to intercept Pod creation requests and automatically inject DNSConfig into newly created Pods.
If the Webhook is disabled, you will need to manually configure and add DNSConfig.
Operating Principle Diagram of NodeLocal DNSCache
- During Pod creation, an Admission Webhook is employed to automatically inject DNSConfig into the Pod.
- In the Pod’s domain name resolution process, the data chain is as follows:
| No. | Description |
|---|---|
| 1 | For Pods with DNSConfig injected, NodeLocal DNSCache listens to the node's IP address by default to handle domain name resolution requests. |
| 2 | If NodeLocal DNSCache lacks a cached response for a resolution request, it forwards the request to CoreDNS or VPC DNS, depending on whether the domain name is intra-cluster or not. |
| 3 | For domain names that cannot be resolved by NodeLocal DNSCache, CoreDNS continues the resolution using the VPC DNS server. |
| 4 | For Pods with DNSConfig injected, local DNS is prioritized for requests. If the local DNS connection fails, the request is forwarded to CoreDNS. However, if the Pod's DNS request library does not support retries after a timeout, only local DNS will be used. |
| 5 | Pods without DNSConfig injection will default to requesting the standard kube-dns service for CoreDNS resolution. |
NodeLocal DNSCache usage
To use NodeLocal DNSCache and redirect traffic to CoreDNS through the local NodeLocal DNSCache DaemonSet proxy, you need to inject DNSConfig configuration into the Pod. Three methods are available:
- Method I: Install the CCE Node Local DNS component and enable DNSConfig injection using the Webhook provided by the component. This is the recommended approach.
- Method II: Manually add DNSConfig to the Pod
- Method III: Modify the kubelet parameters and restart kubelet to apply changes. This method is not recommended due to the potential for service disruptions.
Method I: Automatically inject DNSConfig
Prerequisites:
- Install the CCE NodeLocal DNSCache component in the cluster
The DNSConfig dynamic injection controller simplifies the process by automatically injecting DNSConfig when a Pod is created, removing the need for manual setup. By default, the Webhook listens for Pod creation requests in namespaces labeled with cce.baidu.com/node-local-dns-injection=enabled. You can apply this label to a namespace using the following command:
1kubectl label namespace default cce.baidu.com/node-local-dns-injection=enabledPlease note:
- The above command adds the automatic injection label only to the default namespace, meaning that Pods created in the default namespace will trigger the Webhook for automatic DNSConfig injection. To apply this setup to a different namespace, replace "default" with your desired namespace.
- Under the above conditions, if you need to exempt some Pods (i.e., not inject DNSConfig), you can add the following label to the Pod:
cce.baidu.com/node-local-dns-injection=disabled.
After automatic DNSConfig injection takes effect, newly created Pods will have the following fields added:
1dnsConfig:
2 nameservers:
3 - 169.254.20.10
4 - 172.21.0.10
5 options:
6 - name: ndots
7 value: "3"
8 - name: attempts
9 value: "2"
10 - name: timeout
11 value: "1"
12 searches:
13 - default.svc.cluster.local
14 - svc.cluster.local
15 - cluster.local
16dnsPolicy: NoneDescription:
- nameservers: Configure the IP address corresponding to the ClusterIP of 169.254.20.10 (default) and kube dns; ClusterIP will serve as the backup DNS server address
- searches: Set search domains to ensure that intra-cluster domain names are resolved correctly, defaulting to default.svc.cluster.local, svc.cluster.local and cluster.local
- ndots: The default is 3, and ndots can be appropriately reduced to improve resolution efficiency
- attempts: The default is 2 (number of resolution attempts)
- timeout: The default is 1 (timeout duration)
In Method I, if DNSConfig is not injected into your newly created Pod, check the following conditions (automatic Pod injection is only implemented when all conditions are met):
- The Namespace where the Pod is located contains the label: cce.baidu.com/node-local-dns-injection=enabled
- The namespace where the Pod is located is not: kube-system, cce-monitor or cprom-system
- The Pod does not contain the label: cce.baidu.com/node-local-dns-injection=disabled
- The Pod does not contain scheduling configurations requiring deployment to VirtualKubelet nodes (such as node selection and node affinity)
- The Pod’s hostNetwork is false and DNSPolicy is ClusterFirst or none, or hostNetwork is true and DNSPolicy is ClusterFirstWithHostNet
Method II: Manually add DNSConfig
If Method I is not adopted, you need to manually add DNSConfig. A reference example is as follows:
1apiVersion: v1
2kind: Pod
3metadata:
4 name: test01
5 namespace: default
6spec:
7 containers:
8 - image: registry.baidubce.com/cce-plugin-pro/cce-cni/netshoot:latest
9 command:
10 - sleep
11 - "100000"
12 imagePullPolicy: Always
13 name: net-debug-container
14 dnsPolicy: None
15 dnsConfig:
16 nameservers: ["169.254.20.10","172.21.0.10"]
17 searches:
18 - default.svc.cluster.local
19 - svc.cluster.local
20 - cluster.local
21 options:
22 - name: ndots
23 value: "3"
24 - name: attempts
25 value: "2"
26 - name: timeout
27 value: "1"Please note:
- dnsPolicy must be none
- For descriptions of other parameters, please refer to
Method I
Method III: Configure kubelet startup parameters
You can globally control Pod DNSConfig by modifying kubelet startup parameters. For details, please refer to: Using NodeLocal DNSCache in Kubernetes Clusters.
Component function
- The DNSConfig dynamic injection controller deployment can automatically add DNSConfig configurations to newly created Pods
- Running NodeLocal DNS Cache as a Daemonset on cluster nodes can significantly improve intra-cluster DNS resolution performance
Application scenarios
Deploying NodeLocal DNSCache as a DaemonSet enhances cluster DNS performance.
Usage restrictions
- Cluster version 1.18 or higher
- CCE NodeLocal DNSCache does not support ARM clusters or edge clusters
- Be able to access the cluster using kubectl. For operation steps, refer to: Connect to the Cluster via Kubectl
Install component
There are two ways to install the CCE NodeLocal DNSCache component: via component management or Helm templates. Installation through component management is recommended.
Method I: Install CCE Node Local DNS via component management
- Sign in to the Baidu AI Cloud Official Website and enter the management console.
- Go to Product Services - Cloud Native - Cloud Container Engine (CCE) to access the CCE management console.
- Click Cluster Management > Cluster List in the left navigation bar.
- Click on the target cluster name in the Cluster List page to navigate to the cluster management page.
- On the Cluster Management page, click Component Management.
- In the component management list, select the NodeLocal DNSCache component and click Install.
- Click the OK button to complete the component installation.
Method II: Install CCE NodeLocal DNSCache via Helm template
- Sign in to the Baidu AI Cloud Official Website and enter the management console.
- Select Product Services > Cloud Native > Cloud Container Engine (CCE) to enter the CCE management console.
- Click Helm > Helm Templates in the left navigation bar.
- On the Helm Template page, click Baidu AI Cloud Templates to enter the Template Management page.
- On the Template Management page, select the template named cce-node-local-dns, then click Install.
- On the Install Template page, complete the relevant configuration and click the OK button to complete component installation.
Descriptions of relevant configuration items are as follows:
| Parameters | Required or not | Description | Obtain the path |
|---|---|---|---|
| ClusterDNSAddr | Yes | The IP address corresponding to the ClusterIP of kube-dns in the kube-system namespace enables NodeLocal DNSCache to communicate with CoreDNS for intra-cluster domain name resolution. | Connect to the cluster via kubectl and enter the command kubectl get service kube-dns -n kube-system to obtain the corresponding IP address. You can modify this parameter if you need to specify a different upstream DNS server. No default value; obtained from the cluster. |
| LocalDNSAddr | No | LocalDNSAddr is the local listening IP address selected by cce-node-local-dns | Set to 169.254.20.10 by default (can be modified). |
| searchDomains | No | Search for domain names to ensure that internal domain names in the cluster can be resolved normally | Set to default.svc.cluster.local, svc.cluster.local, cluster.local by default (can be modified). |
| ndots | No | ndots can be appropriately reduced to improve resolution efficiency | Set to 3 by default (can be modified). |
| attempts | No | Number of resolution attempts | Set to 2 by default (can be modified). |
| timeout | No | Timeout duration | Set to 1 by default (can be modified). |
| forbiddenNamespace | No | List of Namespaces where Pod DNSConfig injection is prohibited | Set to kube-system, cprom-system, cce-monitor by default (can be modified). |
Deployment status
When the CCE NodeLocal DNSCache component is installed, the following objects will be deployed in the Kubernetes cluster:
| Resource name | Types |
|---|---|
| node-local-dns | ServiceAccount |
| cce-node-local-dns-cce-reserved | ConfigMap |
| node-local-dns | ConfigMap |
| cce-node-local-dns | ClusterRole |
| cce-node-local-dns | ClusterRoleBinding |
| kube-dns-upstream | Service |
| node-local-dns | Service |
| node-local-dns-webhook | Service |
| node-local-dns | DaemonSet |
| cce-node-local-dns-webhook | Deployment |
| pod-mutating.node-local-dns.cce.baidu.com | MutatingWebhookConfiguration |
Observability
You can monitor the operation of the CCE NodeLocal DNSCache component by scraping the metrics interface exposed by NodeLocal DNSCache.
- Metric data in the cluster can be obtained using this method:
1curl http://<local ip>:9253/metrics # default: 169.254.20.10Introduction to common NodeLocal DNSCache Metrics:
| Metrics | Meaning | Types | Unit |
|---|---|---|---|
| coredns_build_info | CoreDNS version, revision and goversion used for building | Gauge | / |
| coredns_cache_entries | Number of elements in the cache | Gauge | Item |
| coredns_cache_hits_total | Number of cache hits | Counter | Time |
| coredns_cache_misses_total | Number of cache misses | Counter | Time |
| coredns_cache_requests_total | Number of cache requests | Counter | Time |
| coredns_dns_request_duration_seconds | Histogram of time spent on each request per zone | Histogram | Second |
| coredns_dns_request_size_bytes | EDNS0 UDP buffer size (64 K for TCP) | Gauge | Byte |
| coredns_dns_requests_total | Number of DNS requests | Counter | Time |
| coredns_dns_response_size_bytes | Size of returned responses | Gauge | Byte |
| coredns_dns_responses_total | Number of response status codes | Counter | Time |
| coredns_plugin_enabled | Whether a plugin is enabled per server and zone | Gauge | / |
| coredns_reload_failed_total | Number of reload failures | Counter | Time |
Add metric collection task for CCE NodeLocal DNSCache in CProm
Before configuring metric collection rules for Node-Problem-Detector in CProm, ensure the K8S Cluster CCE is associated with a CProm instance and a metric collection task is added. Refer to the operation guide:
Add the following metric collection task:
1job_name: 'localdns'
2scheme: http
3static_configs:
4- targets: [<local ip>:9253']Version records
| Version No. | Cluster version compatibility | Update time | Update content | Impact |
|---|---|---|---|---|
| 1.1.0 | CCE/v1.18/v1.20 | 2023.11.20 | First release | - |
| 1.1.1 | CCE/v1.18/v1.20 | 2023.12.21 | Bug fix: Fix the issue of injecting search domain errors into DNSConfig | - |
| 1.1.2 | CCE/v1.18/v1.20 | 2023.12.22 | Bug fix: Fix Webhook interception rules | - |