Encrypting etcd Data Using KMS

Kubernetes Secret data can be encrypted using keys generated via the Key Management Service (KMS). This document focuses on utilizing the KMS service to encrypt Kubernetes Secret data stored in etcd.

Prerequisites

• Create a customer master key in the KMS console. Only the BAIDU_AES_256 type is supported for the master key. For detailed instructions, refer to Key Management Service (KMS)
• Configure KMS identity and access management. For configuration steps, see Configure KMS Identity and Access Management
• Using the KMS service incurs fees; no additional fees are charged for CCE. For details, refer to KMS Product Pricing

Configure KMS identity and access management and obtain AK/SK

Create IAM user

1. After logging into the root account, select Identity and Access Management from the console to access the user management page.

2. In the left navigation bar, click on User Management, then on the User Management - IAM User list page, select Create IAM User and check the Programmatic Access option.

3. Download and securely save the AK/SK credentials, as they will be required during KMS plugin deployment.

Configuration Policy

1. To create a policy, navigate to Policy Management in the left menu, click Create Policy on the new page, and choose the Create via Visual Editor option.

2. Add permissions by clicking Add Permissions. Select Key Management Service (KMS) and Encryption/Decryption, then fill in the required fields and confirm.

Configure IAM user permissions

Go to User Management > IAM Users in the left navigation bar to view the user list. Locate the desired IAM user, click Edit Permissions, select the appropriate policy, and confirm.

Install the k8s-cloudkms-plugin

The following operations must be performed on each master node in sequence

Download the binary file for the k8s-cloudkms-plugin.

1wget -O /opt/kube/bin/k8s-cloudkms-plugin https://cce-plugin.bj.bcebos.com/kms-plugin/v1.0.0/k8s-cloudkms-plugin
2chmod +x /opt/kube/bin/k8s-cloudkms-plugin

Create a configuration file for k8s-cloudkms-plugin and fill in necessary information (it is recommended to use the AK/SK saved earlier for the AK/SK field). The configuration content is as follows:

1vim /etc/systemd/system/k8s-cloudkms-plugin.service
2[Unit]
3Description=Kubernetes kms plugin
4After=network.target
5[Service]
6ExecStart=/opt/kube/bin/k8s-cloudkms-plugin \
7--region={{region}} \
8--key-id={{Kms Key ID}} \
9--access-key={{Access Key ID}} \
10--secret-key={{Secret Access Key}} \
11--kms-endpoint={{Kms Endpoint}}
12Restart=always
13Type=simple
14LimitNOFILE=65536
15[Install]
16WantedBy=multi-user.target

Configuration for the Beijing region is as follows:

1[Unit]
2Description=Kubernetes kms plugin
3After=network.target
4[Service]
5ExecStart=/opt/kube/bin/k8s-cloudkms-plugin \
6--region=bj \
7--key-id=your-kms-key-id \
8--access-key=your-ak-id \
9--secret-key=your-sk-id \
10--kms-endpoint=https://bkm.bj.baidubce.com
11Restart=always
12Type=simple
13LimitNOFILE=65536
14[Install]
15WantedBy=multi-user.target

Start k8s-cloudkms-plugin

1systemctl daemon-reload && systemctl enable k8s-cloudkms-plugin.service && systemctl restart k8s-cloudkms-plugin.service

Create a sample kms-encryption-config.yaml file as follows:

1vim /etc/kubernetes/kms-encryption-config.yaml
2kind: EncryptionConfiguration
3apiVersion: apiserver.config.k8s.io/v1
4resources:
5  - resources:
6    - secrets
7    providers:
8    - kms:
9        name: cceKMSPlugin
10        endpoint: unix:///var/run/kmsplugin/socket.sock
11        cachesize: 1000
12        timeout: 3s
13    - identity: {}

Add a startup parameter for kube-apiserver: Use the --encryption-provider-config parameter to enable etcd on-disk encryption:

• When kube-apiserver is started as a binary and managed by systemd, modify as follows

1# Check if it is started as a binary (a running process should be visible)
2systemctl status kube-apiserver.service
3# Modify the kube-apiserver startup file
4vim /etc/systemd/system/kube-apiserver.service
5# Add the new parameter encryption-provider-config
6--encryption-provider-config=/etc/kubernetes/kms-encryption-config.yaml \
7# Restart kube-apiserver
8systemctl daemon-reload && systemctl restart kube-apiserver.service

• When kube-apiserver is deployed in containerized mode, modify as follows

1# Check if it is deployed in containerized mode (an apiserver pod should be visible)
2kubectl get pod -n kube-system | grep kube-apiserver
3# Modify the kube-apiserver manifests file
4vim /etc/kubernetes/manifests/kube-apiserver.yaml
5# Add the new startup parameter encryption-provider-config
6- --encryption-provider-config=/etc/kubernetes/kms-encryption-config.yaml
7# Mount the kmsplugin into the container; add kmsplugin to volumeMounts and volumes
8volumeMounts:
9  - mountPath: /var/run/kmsplugin
10    name: kmsplugin
11    
12volumes:
13  - hostPath:
14      path: /var/run/kmsplugin
15      type: Directory
16    name: kmsplugin

Perform the described operations on each master node sequentially.

Verification

1. Create a new secret in the cluster

   Plain Text Copy

   1kubectl create secret generic secret1 -n default --from-literal=mykey=mydata

2. Run the command below to verify that the secret has been decrypted correctly.

   Plain Text Copy

   1# The output result should be mydata
   2kubectl get secret secret1 -o=jsonpath='{.data.mykey}' | base64 -d

3. (Optional) Verify if the data in etcd is encrypted: Sign in to a master node. Replace the IP in the following command with the machine’s IP and execute it:

   Plain Text Copy

   1ETCDCTL_API=3 /opt/etcd/bin/etcdctl --cacert=/etc/etcd/ssl/ca.pem --cert=/etc/etcd/ssl/etcd.pem --key=/etc/etcd/ssl/etcd-key.pem --endpoints=https://{hostIP}:2379 get  /registry/secrets/default/secret1 | hexdump -C
   2# If the output contains “cceKMSPlugin” and no plaintext data, the encryption is successful

4. To encrypt all secrets for on-disk storage, execute the corresponding command:

   Plain Text Copy

   1kubectl get secrets --all-namespaces -o json| kubectl replace -f -

Reference documentation

• Key Management Service (KMS)
• Reference document for implementing kms-provider
• Kubernetes encrypt-data principle