Best Practices for CCE Support Periodic and Timing Scaling Deployment
I. Introduction to Timing and Periodic Scaling
For some predictable business situations, you can set the timing scaling tasks in advance. You can scale up the working container before the business peak comes to reduce the workload of operation and maintenance personnel. Currently, the CCE supports the setting of scaling tasks for Deployment and StatefulSet and meanwhile supports the combination use of scaling and HPA to greatly expand the application scenarios of elastic scaling.
II. Usage Guide
Premise: Create CCE cluster and cluster version >1.13.x
In the created CCE cluster, follow the steps as below:
1. Create CRD yaml
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: cronhpas.cce.io
spec:
group: cce.io
version: v1
names:
kind: CronHPA
listKind: CronHPAList
plural: cronhpas
singular: cronhpa
scope: Namespaced 2. Create related resources of cronHPA_controller.
apiVersion: v1
kind: ServiceAccount
metadata:
name: cce-cronhpa-controller
namespace: kube-system
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: cce-cronhpa-controller
namespace: kube-system
subjects:
- kind: ServiceAccount
name: cce-cronhpa-controller
namespace: kube-system
roleRef:
kind: ClusterRole
name: cce-cronhpa-controller
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: cce-cronhpa-controller
namespace: kube-system
rules:
- apiGroups:
- ""
resources:
- deployments
- statefulsets
- events
verbs:
- '*'
- apiGroups: ["extensions", "apps"]
resources:
- deployments
- deployments/scale
- statefulsets
- replicasets
verbs:
- '*'
- apiGroups:
- autoscaling
resources:
- horizontalpodautoscalers
verbs:
- '*'
- apiGroups:
- apiextensions.k8s.io
resources:
- customresourcedefinitions
verbs:
- '*'
- apiGroups:
- cce.io
resources:
- '*'
verbs:
- '*'
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: cron-hpa-controller
namespace: kube-system
labels:
app: cron-hpa-controller
spec:
replicas: 1
selector:
matchLabels:
app: cron-hpa-controller
template:
metadata:
labels:
app: cron-hpa-controller
spec:
serviceAccountName: cce-cronhpa-controller
containers:
- name: cron-hpa-controller
image: hub.baidubce.com/jpaas-public/cce-cronhpa-controller:latest
imagePullPolicy: Always
command:
- ./usr/local/bin/cce-cronhpa-controller
resources:
limits:
memory: 512Mi
cpu: 200m
requests:
memory: 200Mi
cpu: 100m
volumeMounts:
- name: host-time
mountPath: /etc/localtime
readOnly: true
volumes:
- name: host-time
hostPath:
path: /etc/localtime At this time, you build the cronHPA deployment environment, and then you can create the corresponding instance resources.
3. Create cronHPA resources and their cited subresources.
The cited sub-object is deployment.
The CronHPA and Deployment resources are created here.
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: cronhpa-test
spec:
replicas: 1
template:
metadata:
name: cronhpa-test
labels:
name: cronhpa-test
spec:
containers:
- name: nginx
image: hub.baidubce.com/jpaas-public/nginx-alpine-go
ports:
- containerPort: 80
---
apiVersion: cce.io/v1
kind: CronHPA
metadata:
name: cronhpa-sample
namespace: default
spec:
scaleTargetRef:
apiVersion: extensions/v1beta1
kind: Deployment
name: cronhpa-test
excludeDates:
- "*/2 * * * *"
crons:
- name: "scale-down"
schedule: "*/1 * * * *"
targetSize: 2
runOnce: true
- name: "scale-up"
schedule: "*/2 * * * *"
targetSize: 3
runOnce: false scaleTargetRef: Non-null, the field identifies the resource object specified by the cronhpa, namely, scaling object. The resource object can be deployment and statefulset.
excludeDates: One cron character string array. The cron expression group computes the time set that the scaling is not executed next time according to the current time and LastScheduleTime, so as to realize the date filtering feature (note: Here the relationship among multiple crons is "or", namely, each cron represents one filtering time).
crons: Below the field is the set of cron tasks
cron: name: Character string, non-null. As the unique identification of the cron (set cron name, guarantee that the names of all crons in the same crons array are different).
targetSize: int32, non-null. Set it as the number of duplicates of subresources cited as the user expects.
schedule: Character string, non-null. The cron expression group computes the time set that the scaling is not executed next time according to the current time and LastScheduleTime.
runOnce: The Boolean type may be empty (false by default). The true means whether the cron is executed once only.
Note: Multiple crons may operate the same resource object at the same time. At this time, you can select the maximum cron task of targetSize for execution, so as to guarantee that the business is not impacted.
The cited sub-object is HorizontalPodAutoscaler.
The CronHPA, HorizontalPodAutoscaler and Deployment resources are created here.
Note This is mainly to solve the problem that CronHPA and HPA cite the same Deployment. The user shouldn't let CronHPA and HPA cite the same Deployment. But CronHPA cites HPA and HPA cites Deployment. In this way, CronHPA and HPA can be used simultaneously for the same Deployment.
apiVersion: apps/v1
kind: Deployment
metadata:
name: hpa-deploy
labels:
app: hpa-deploy
spec:
replicas: 5
selector:
matchLabels:
app: hpa-deploy
template:
metadata:
labels:
app: hpa-deploy
spec:
containers:
- name: hpa-deploy
image: hub.baidubce.com/jpaas-public/hpa:hpa-example
resources:
limits:
cpu: 100m
memory: 300Mi
requests:
cpu: 10m
memory: 30Mi
ports:
- containerPort: 80
---
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
metadata:
name: cronhpa-hpa-test
namespace: default
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: hpa-deploy
minReplicas: 1
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
---
apiVersion: cce.io/v1
kind: CronHPA
metadata:
name: cronhpa-hpa-sample
namespace: default
spec:
scaleTargetRef:
apiVersion: autoscaling/v2beta2
kind: HorizontalPodAutoscaler
name: cronhpa-hpa-test
crons:
- name: "scale-up"
schedule: "*/1 * * * *"
targetSize: 11 scaleTargetRef: Under this situation, the kind is HorizontalPodAutoscaler.
The meanings of other fields are the same as those of deployment.
This is mainly to be compatible with HPA and the timing scaling components to ensure the two can cite the same resource object. Currently, Baidu AI Cloud supports the following conflict resolution and compatibility.
| HPA(MIN/MAX) | cronHPA(targetSize) | deploy(currentSize) | result | solution |
|---|---|---|---|---|
| 5/10 | 4 | 5 | hpa( 4/ 10) deployment 5 | When the set time is later than the current time, retain the time set for the current duplicate lower than the HPA lower limit, and modify the HPA lower limit. |
| 5/10 | 11 | 5 | hpa(11/11) deployment 11 | When the set time is earlier than the current time, retain the time set for the timing duplicate lower than the HPA upper limit, and modify the HPA upper limit. |
| 1/5 | 2 | 2 | hpa(1/5) deployment 2 | When the set time and the current time are consistent, no changes may be made. |
| 1/5 | 2 | 3 | hpa(1/5) deployment 3 | When the current time is earlier than the set time, retain the current duplicate. |
| 1/5 | 3 | 2 | hpa( 3/5) deployment 3 | When the set time is earlier than the current time, retain the time set for the timing duplicate lower than the HPA lower limit, and modify the HPA lower limit. |
The only compatibility idea is to maintain the maximum duplicates of deploy, and ensure the deploy is in the status of maximum duplicates.
III. Common Cases
Setting format of cron
1、 Execute once at the 0 minute of every hour on every day in every month: 0 * * * *
2、 Execute once every other 3 hours from 6:00 to 12:00 AM each day in the 12 months. 0 6-12/3 12
3、 Execute once at 5:00 pm from Monday to Friday: 0 17 * * 1-5
4、 Execute once at 0:20, 2:20, 4:20 on every day in each month: 20 0-23/2 * * *
IV. FAQ
1. Time granularity of controller synchronization
The CronHPA controller synchronizes all CronHPA resources in the cluster every other 1 minute. (the minimum granularity of cron is minute)
2. Whether to support multiple duplicates of controller
Support
3. Why the set cron task doesn't become valid
In the kube-system command space, view whether the cce-cronhpa- controller components operate normally. View the log of cce-cronhpa-controller component and whether the "too many missed start time (> 100)" error is reported. The controller may accumulate too many unprocessed historical tasks. Submit a ticket for solution