Gated Launch
Last Updated：2020-09-21
This article introduces how to use the Ingress feature of Baidu AI Cloud container service, and realize the blue-green release.
Background Information
The gray and blue-green releases aim to create a production environment for the new version which is completely consistent with the old version. Under the premise of not impacting the old version, some traffic is switched to the new version according to certain rules. After the trial operation of the new version for a period without any problems, the total traffic of the user is migrated to the new version from the old version.
And the A/B test is a kind of gray release mode. Some users continue to use the services of the old version, and the traffic of some users is switched to the new version. If the new version runs stably, all the users are migrated to the new version step by step.
Ingress-Nginx Annotation Introduction
Based on Nginx Ingress Controller, CCE realizes the project gateway, and serves as the external traffic ingress of the project and the reverse proxy of each service in the project. And Ingress-Nginx supports the configuration of Ingress Annotations to realize the gray release and test of different scenarios to satisfy the canary release, blue-green deployment and A/B test, and other business scenarios.
The Nginx Annotations supports the following 4 Canary rules:
- nginx.ingress.kubernetes.io/canary-by-header: The traffic segmentation based on Request Header is applicable to the gray release and A/B test. When the Request Header is set as always, the requests are delivered to the Canary version all the time; when the Request Header is set as never, the request is not sent to the Canary ingress; for any other Header values, the Header is ignored, and the requests and other canary rules are compared in priority through the priority.
- nginx.ingress.kubernetes.io/canary-by-header-value: The value of the Request Header to be matched is used to give a notice to Ingress to route the request to the service specified in Canary Ingress. When the Request Header is set as this value, it is routed to the Canary Ingress. This rule allows the users to customize the value of the Request Header, and should be used together with the previous annotation (namely: canary-by-header).
- nginx.ingress.kubernetes.io/canary-weight: The traffic segmentation based on the service weight is applicable to the blue-green deployment, the weigh range is within 0-100. The request is routed to the service specified in Canary Ingress in percentage. The weight of 0 means the canary rule can't send any request to the service of Canary Ingress. The weight of 100 means all requests are sent to the Canary Ingress.
- nginx.ingress.kubernetes.io/canary-by-cookie: The traffic segmentation based on Cookie is applicable to the gray release and A/B test. It is used to give a notice to Ingress to route the request to cookie of the service specified in Canary Ingress. When the cookie value is set as always, it is routed to the Canary Ingress; when the cookie value is set as never, the request is not sent to the Canary ingress; for any other values, the cookie is ignored, and the requests and other canary rules are compared in priority.
canary Rule priority: canary-by-header - > canary-by-cookie - > canary-weight
Installation nginx-ingress-controller
# yaml For the document contents, refer to the Appendix.
kubectl apply -f ingress-nginx.yaml
kubectl apply -f ingress-nginx-service.yaml Deployment production task
1.Create the production application resources
kubectl apply -f production.yaml -n canary-demo apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: production
spec:
replicas: 1
selector:
matchLabels:
app: production
template:
metadata:
labels:
app: production
spec:
containers:
- name: production
image: hub.baidubce.com/jpaas-public/echoserver:1.10
ports:
- containerPort: 8080
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
---
apiVersion: v1
kind: Service
metadata:
name: production
labels:
app: production
spec:
ports:
- port: 80
targetPort: 8080
protocol: TCP
name: http
selector:
app: production 2.Create the application route of Production version (Ingress)
kubectl apply -f production.ingress.yaml -n canary-demo apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: production
annotations:
kubernetes.io/ingress.class: nginx
spec:
rules:
- host: cce.canary.io
http:
paths:
- backend:
serviceName: production
servicePort: 80 3.Local machine access application:
Binding hosts: vi /etc/hosts
106.12.7.210 cce.canary.io
curl cce.canary.io Following access success
Create the Tasks of Canary Version
1.Create the application resources of canary version
kubectl apply -f canary.yaml -n canary-demo apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: canary
spec:
replicas: 1
selector:
matchLabels:
app: canary
template:
metadata:
labels:
app: canary
spec:
containers:
- name: canary
image: hub.baidubce.com/jpaas-public/echoserver:1.10
ports:
- containerPort: 8080
env:
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
- name: POD_IP
valueFrom:
fieldRef:
fieldPath: status.podIP
---
apiVersion: v1
kind: Service
metadata:
name: canary
labels:
app: canary
spec:
ports:
- port: 80
targetPort: 8080
protocol: TCP
name: http
selector:
app: canary 2.Create the application route of canary version based on weight (Ingress)
kubectl apply -f canary.ingress.yaml -n canary-demo apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: canary
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/canary: "true"
nginx.ingress.kubernetes.io/canary-weight: "30"
spec:
rules:
- host: cce.canary.io
http:
paths:
- backend:
serviceName: canary
servicePort: 80 3.Verification of access application domain name
for i in $(seq 1 10); do curl cce.canary.io| grep Hostname ; done As shown in the figure below, some traffic flows into canary
After the traffic segmentation of the Canary version of application based on weight (30%), the probability of accessing Canary version is close to 30%, and the traffic ratio may float in a small range. This is a normal phenomenon.
Appendix: ingress-nginx Related yaml Files
- ingress-nginx.yaml
apiVersion: v1
kind: Namespace
metadata:
name: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: nginx-ingress-clusterrole
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
- "networking.k8s.io"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
- "networking.k8s.io"
resources:
- ingresses/status
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-role
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "<election-id>-<ingress-class>"
# Here: "<ingress-controller-leader>-<nginx>"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: nginx-ingress-role-nisa-binding
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-role
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: nginx-ingress-clusterrole-nisa-binding
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nginx-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: ingress-nginx
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-ingress-controller
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
template:
metadata:
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
spec:
# wait up to five minutes for the drain of connections
terminationGracePeriodSeconds: 300
serviceAccountName: nginx-ingress-serviceaccount
nodeSelector:
kubernetes.io/os: linux
containers:
- name: nginx-ingress-controller
image: hub.baidubce.com/jpaas-public/nginx-ingress-controller:0.30.0
args:
- /nginx-ingress-controller
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --publish-service=$(POD_NAMESPACE)/ingress-nginx
- --annotations-prefix=nginx.ingress.kubernetes.io
securityContext:
allowPrivilegeEscalation: true
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
# www-data -> 101
runAsUser: 101
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
protocol: TCP
- name: https
containerPort: 443
protocol: TCP
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
lifecycle:
preStop:
exec:
command:
- /wait-shutdown
---
apiVersion: v1
kind: LimitRange
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
limits:
- min:
memory: 90Mi
cpu: 100m
type: Container - ingress-nginx-service.yaml
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
namespace: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
spec:
externalTrafficPolicy: Cluster
type: LoadBalancer
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
ports:
- name: http
port: 80
protocol: TCP
targetPort: http
- name: https
port: 443
protocol: TCP
targetPort: https