Configure Collection Rules Using CRD
Updated at：2025-10-27
Operation scenarios
You can not only configure log collection on the console but also configure it through custom resource definitions (CRDs). CRD supports collecting container stdout and internal logs of containers. The log source can collect both host and container logs, supports multiple log collection formats, and supports delivering to different consumer ends such as BLS and BES.
Prerequisites
- Successfully create CCE cluster.
- If the Baidu AI Cloud Log Service (BLS) has been activated, you can log in to the Baidu AI Cloud Log Service (BLS) Console to confirm its status.
CRD introduction
The specific meanings of each field defined in the CRD are as follows:
CRD resource type
- LogConfig
LogConfig definition
| Field | Description |
|---|---|
apiVersion string |
cce.baidubce.com/v1 |
kind string |
LogConfig |
metadata ObjectMeta |
For details, see Kubernetes API metadata. |
spec LogConfigSpec |
See below for details |
status LogConfigStatus |
Collection task status and associated collector ID |
LogConfigSpec definition
| Field | Description |
|---|---|
srcConfig |
See below for details |
dstConfig |
See below for details |
SrcConfig definition
| Field | Description |
|---|---|
srcType SrcType |
srcType, optional values: host or container (required) |
logType ContainerLogType |
logType is only applicable when the srcType field is container. Optional values: stdout and internal, representing stdout logs and internal logs of container, respectively (required) |
srcDir string |
srcDir, log collection directory (note: it is a directory). When srcType=container and logType=stdout, srcDir does not need to be specified (optional) |
matchPattern string |
Log file matching rules under matchPattern srcDir (optional) |
ignorePattern string |
Log file ignore rules under ignorePattern srcDir (optional) |
timeFormat string |
timeFormat used for date resolution of original file paths when delivering BOS (optional) |
ttl integer |
ttl agent, log collection time range (unit: days(required) |
useMultiline boolean |
useMultiline, options for multi-line mode use (optional) |
multilineRegex string |
multilineRegex, first-line pattern in multi-line mode (optional) |
recursiveDir boolean |
recursiveDir, options for recursively collect all files that match matchPattern under srcDir, including subdirectories (optional) |
logTime LogTimeType |
logTime (log time), options: system and logTime, indicating the use of system time and the use of log time respectively (optional) |
timestampKey string |
timestampKey, specify the resolved field as the log timestamp (optional) |
dateFormat string |
dateFormat, specify the time resolution format for the timestamp field (for the format, refer to https://docs.oracle.com/javase/7/docs/api/java/text/SimpleDateFormat.html "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'Z” (optional) |
filterExpr string |
filterExpr, log matching expression -only logs conforming to the rule will be collected (optional) |
processType ProcessType |
processType resolution type (optional) |
processConfig ProcessConfig |
processConfig, parameters for corresponding processType, see below for details (optional) |
matchLabels KVPair array |
matchLabels container environment matching labels (the label corresponds to the label in docker inspect, not the label on the pod) (optional) |
ignoreLabels KVPair array |
ignoreLabels, labels ignored in the container environment, same as above (optional) |
matchEnvs KVPair array |
matchEnvs, container environment matching env (optional) |
ignoreEnvs KVPair array |
matchEnvs, container environment matching env that is ignored (optional) |
Either matchLabels or matchEnvs must be filled in
DstConfig definition
Set the destination location for log data delivery. Currently, two options are available: LogStore and BES. The configuration details for each are as follows:
Set the logstore as the destination location
| Field | Description |
|---|---|
dstType DstType |
dstType, destination location storage type, currently support BLS (required) |
logStore string |
logStore, BLS logstore name (required) |
retention integer |
retention, log storage duration, range: 1-90, unit is day (required) |
rateLimit integer |
Log upload bandwidth limit, range: 1-100, unit: MB (required) |
Set BES (Elasticsearch) as the destination location
| Field | Description |
|---|---|
dstType DstType |
dstType, destination location storage type, currently support BES (required) |
besClusterID string |
BES cluster ID, which must be in the same region (required) |
besUser string |
BES cluster login username (required) |
besPasswd string |
BES cluster login password (required) |
besIndexPrefix string |
Custom index prefix: When index rolling is enabled, the index name in the BES cluster follows the format “index prefix + collection date”; when disabled, the index name includes only the prefix. The collection date corresponds to the date the data is written into BES, formatted as YYYY-MM-DD (required). |
besIndexRolling string |
Specify the frequency at which the BES cluster generates a new index automatically. Options include none (no rolling index generation), daily (generated at 00:00 each day), weekly (generated at 00:00 every Monday), or monthly (generated at 00:00 on the 1st of each month) (required). |
besIsPwChange boolean |
Only support inputting true (required) |
ProcessConfig definition
| Field | Description |
|---|---|
regex string |
Regex regular rules (optional) |
separator string |
Separator delimiter (optional) |
quote string |
quote: Specify quote characters for separator scenarios, options: blank, double quote ("), single quote ('), and custom (optional) |
sampleLog string |
sampleLog: Log processing example, used to configure keys and dataType in console after processing (optional) |
keys string |
keys: column names for resolution results; @message is a system-reserved word and cannot be set as a key (optional) |
dataType string |
dataType, data type corresponding to each column in the resolution result, support string/int/float/bool, and it must correspond to keys one to one (optional) |
discardOnFailure boolean |
discardOnFailure, options for log discard upon resolution failure (true: discard; false: return original value (optional) |
KeepOriginal boolean |
keepOriginal, options for log retention; true: retain the original logs in the @message field of kafka and in the @raw field of BLS; false: do not retain the original logs upon successful resolution (optional) |
CRD example
- Note: Due to BLS limitations, the default maximum count of log files that can be processed simultaneously by a single node is currently 10
Container Stdout CRD configuration example
Collect logs from all containers across all namespaces
YAML
1apiVersion: cce.baidubce.com/v1
2kind: LogConfig
3metadata:
4 name: all-container-stdout-log
5 namespace: kube-system
6spec:
7 srcConfig:
8 srcType: container
9 logType: stdout
10 ttl: 3
11 matchLabels:
12 - key: io.kubernetes.container.name
13 value: .*
14 - key: io.kubernetes.pod.namespace
15 value: .*
16 dstConfig:
17 dstType: BLS
18 logStore: container-stdout-log
19 retention: 10
20 rateLimit: 10Collect logs from all containers across specified namespaces
YAML
1apiVersion: cce.baidubce.com/v1
2kind: LogConfig
3metadata:
4 name: kube-system-container-stdout-log
5 namespace: kube-system
6spec:
7 srcConfig:
8 srcType: container
9 logType: stdout
10 ttl: 3
11 matchLabels:
12 - key: io.kubernetes.container.name
13 value: .*
14 - key: io.kubernetes.pod.namespace
15 value: kube-system
16 dstConfig:
17 dstType: BLS
18 logStore: kube-system-container-stdout-log
19 retention: 10
20 rateLimit: 10Collect logs from specified containers in specified namespace
YAML
1apiVersion: cce.baidubce.com/v1
2kind: LogConfig
3metadata:
4 name: kube-state-metrics-log
5 namespace: kube-system
6spec:
7 srcConfig:
8 srcType: container
9 logType: stdout
10 ttl: 3
11 matchLabels:
12 - key: io.kubernetes.container.name
13 value: kube-state-metrics
14 - key: io.kubernetes.pod.namespace
15 value: kube-system
16 dstConfig:
17 dstType: BLS
18 logStore: kube-state-metrics-log
19 retention: 10
20 rateLimit: 10Container file path CRD configuration example
Collect logs from the file paths within specified containers in specified namespace
Note: Container internal file paths, for example, /usr/local/tomcat/logs/ requires to mount emptyDir
YAML
1apiVersion: cce.baidubce.com/v1
2kind: LogConfig
3metadata:
4 name: tomocat-log
5 namespace: default
6spec:
7 srcConfig:
8 srcType: container
9 logType: internal
10 srcDir: /usr/local/tomcat/logs/ # container internal file path, note that /usr/local/tomcat/logs/ requires to mount emptyDir
11 matchPattern: catalina.*.log # container internal log file name (supports regular)
12 ttl: 3
13 matchLabels:
14 - key: io.kubernetes.container.name
15 value: tomcat
16 - key: io.kubernetes.pod.namespace
17 value: default
18 dstConfig:
19 dstType: BLS
20 logStore: tomocat-log
21 retention: 10
22 rateLimit: 10Tomcat deployment reference example
YAML
1apiVersion: apps/v1
2kind: Deployment
3metadata:
4 name: tomcat
5spec:
6 selector:
7 matchLabels:
8 app: tomcat
9 replicas: 4
10 template:
11 metadata:
12 labels:
13 app: tomcat
14 spec:
15 containers:
16 - name: tomcat
17 image: "tomcat:7.0"
18 volumeMounts:
19 - name: tomcat-log
20 mountPath: /usr/local/tomcat/logs # mounts emptyDir
21 volumes:
22 - name: tomcat-log # log
23 emptyDir: {}Node file path CRD configuration example
Collect log files under specified directories on nodes
YAML
1apiVersion: cce.baidubce.com/v1
2kind: LogConfig
3metadata:
4 name: host-messages
5 namespace: kube-system
6spec:
7 srcConfig:
8 srcType: host
9 srcDir: /logbeat_host/var/log # Note: host logs require to add the /logbeat_host prefix
10 matchPattern: messages # collect message log files
11 ttl: 3
12 dstConfig:
13 dstType: BLS
14 logStore: host-messages
15 retention: 10
16 rateLimit: 10