CCE Node Problem Detector Description

Component introduction

In real Kubernetes cluster usage, nodes may experience various issues that impact overall service availability. Additionally, by default, the Kubernetes control plane does not promptly detect node anomalies, leading to instances being scheduled to unavailable nodes. Cluster users often are not immediately aware of node or instance failures, delaying corrective actions. When such failures occur, users expect the platform to enable automatic fault recovery, such as migrating instances from fault nodes.

Node-Problem-Detector enhances the cluster's node fault detection capabilities. When users install this component in the cluster, it operates as a DaemonSet to monitor real-time node anomalies and reports results to the upstream Kube-APIServer.

Component function

• Provide node fault detection capabilities

• Supported fault reporting methods include

  • Node condition: Pod may not be able to run on this node
  • Event: Temporary issues affecting the node, but meaningful for system diagnosis
  • Metrics: Prometheus metrics are generated for both of the above scenarios, available for collection and display

Limitations

• Cluster version 1.18.9 or higher

Install component

1. Sign in to the Baidu AI Cloud official website and enter the management console.
2. Go to Product Services - Cloud Native - Cloud Container Engine (CCE) to access the CCE management console.
3. Click on Cluster Management > Cluster List in the left navigation bar.
4. Click on the target cluster name in the Cluster List page to navigate to the cluster management page.
5. On the Cluster Management page, navigate to Component Management.
6. In the component management list, find the CCE Node Problem Detector component and click Install.
7. Click the OK button to finalize the component installation.

View cluster health check status

After installing the component, you can view fault statuses on the node health check interface.

1. Sign in to the Baidu AI Cloud Official Website and enter the management console.
2. Go to Product Services - Cloud Native - Cloud Container Engine (CCE) to access the CCE management console.
3. Click on Cluster Management > Cluster List in the left navigation bar.
4. Click on the target cluster name in the Cluster List page to navigate to the cluster management page.
5. On the Cluster Management page, go to Node Management > Worker to access the node management interface.
6. Click the Health Check icon to display the node’s health check status:

7. Review the health check statuses. Currently, only NetworkUnavailable and Ready states affect node availability.

Fault status

Node Conditions

After installing CCE-Node-Problem-Detector, the following conditions will be added to nodes:

Introduction to the EBC model fault detection

For elastic baremetal compute (EBC), Node-Problem-Detector integrates with Baidu AI Cloud’s hardware awareness component HAS-agent, adding hardware health detection capabilities for GPU/RDMA network interface cards/CPUs/memory, etc.

Prerequisites

1. The Baidu AI Cloud’s hardware awareness component HAS-agent is installed on the machine.

GPU detection

CPU/memory detection

Network interface card detection

Implement a fault alarm using monitor metrics

Introduction to Node-Problem-Detector metrics

Node-Problem-Detector exposes the following Prometheus metrics:

The LabelSet included in the metric is as follows:

1. instance: Fault node
2. namespace: The namespace where the NPD instance is deployed
3. type: Fault type, corresponding to Node Condition Type
4. reason: Fault reason, corresponding to Node Condition Reason
5. region: The region where the node is located

Users can configure Prometheus monitors and alerts based on the above metrics as needed;

Add metric collection task for Node-Problem-Detector in CProm

Before adding metric collection rules for Node-Problem-Detector in CProm, you need to associate the K8S Cluster CCE with a CProm instance and add a metric collection task first. Operation references:

1. Associate CProm Instance with K8S Cluster CCE
2. Add Collection Task in CProm Instance

Add the following metric collection task:

1job_name: node-problem-detector
2scrape_interval: 3s
3kubernetes_sd_configs:
4  - role: endpoints
5relabel_configs:
6  - source_labels: [ __meta_kubernetes_service_annotation_npd_prometheus_scrape ]
7    action: keep
8    regex: true
9  - source_labels: [ __meta_kubernetes_service_annotation_npd_prometheus_scheme ]
10    action: replace
11    target_label: __scheme__
12    regex: (https?)
13  - source_labels: [ __meta_kubernetes_service_annotation_npd_prometheus_path ]
14    action: replace
15    target_label: __metrics_path__
16    regex: (.+)
17  - source_labels: [ __address__, __meta_kubernetes_service_annotation_npd_prometheus_port ]
18    action: replace
19    target_label: __address__
20    regex: ([^:]+)(?::\d+)?;(\d+)
21    replacement: $1:$2
22  - source_labels: [ __meta_kubernetes_endpoint_node_name ]
23    action: replace
24    target_label: node
25  - source_labels: [ __meta_kubernetes_namespace ]
26    action: replace
27    target_label: namespace
28  - source_labels: [ __meta_kubernetes_service_name ]
29    action: replace
30    target_label: service
31  - source_labels: [ __meta_kubernetes_pod_name ]
32    action: replace
33    target_label: pod

Version records