P1. What is Kustomize?
Helm supports advanced editing of each location of yaml on a given environment . Kustomize has the strength of supporting multiple environments (prod, stag, dev1, dev2…..) through the use of overlays. (Also, we can still combine kustomize and helm together)
- Setting
1 2 3 4 5 6 7 8 | # curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash # mv kustomize /usr/local/bin/ #----Thêm bash-completion cho kustomize---- # yum install bash-completion # kustomize completion bash > /etc/bash_completion.d/kustomize > thoát phiên ssh và vào lại. |
- Kustomize’s directory structure
- Instructions for creating kustomization.yaml
1 2 3 4 5 6 7 8 | # mkdir -p 01.resouce/base # cd 01.resouce/base # kustomize init #(Lệnh này sẽ tạo ra file kustomization.yaml mặc định) # cat kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization |
YAML sample full article, you can refer here https://github.com/worldhello12/kustomize-sample
P2: Resources
Definition: Resources consist of 1 or more files .yaml k8s > As input input for kustomize stir-fry.
Example of using Resources 
- Edit the file kustomization.yaml just init
1 2 3 4 5 6 7 8 | # vim kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - api-depl.yaml - api-service.yaml |
- Prepare deployment and sample service: api-depl.yaml and api-service.yaml
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 | # cd 01.resouce/base # kubectl create deployment nginx-api --image=nginx:alpine --replicas=3 --dry-run=client -o yaml > api-depl.yaml # vim api-service.yaml apiVersion: v1 kind: Service metadata: creationTimestamp: null labels: app: nginx-api name: nginx-api spec: ports: - port: 80 protocol: TCP targetPort: 80 nodePort: 30181 selector: app: nginx-api type: NodePort status: loadBalancer: {} |
- Run Test build Kustomize
1 2 3 4 5 6 7 8 9 | # kustomize build . > Kết quả ta có yaml final để chạy k8s # kustomize build . | kubectl apply -f - service/nginx-api created deployment.apps/nginx-api created # kustomize build . | kubectl delete -f - |
P3: Transformers common
(reference: https://github.com/kubernetes-sigs/kustomize/tree/master/examples/transformerconfigs )
Kustomize supports transformers converting the following types:
1 2 3 4 5 6 | - Sửa (images) - Thêm (annotations) - Thêm (labels) - Thêm (namespace) - Thêm (prefix/suffix) (đầu/cuối) |
Let’s go into the transformer example:
Step 1. Prepare deployment and sample service (api-depl.yaml and api-service.yaml)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 | # mkdir -p 02.transformer/base # cd 02.transformer/base # kubectl create deployment web-frontend --image=httpd:alpine --replicas=3 --dry-run=client -o yaml > api-depl.yaml # vim api-service.yaml apiVersion: v1 kind: Service metadata: creationTimestamp: null labels: app: nginx-api name: nginx-api spec: ports: - port: 80 protocol: TCP targetPort: 80 nodePort: 30181 selector: app: nginx-api type: NodePort status: loadBalancer: {} |
Step 2. Edit kustomization.yaml
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 | [tuanda@master-node ]$ kustomize init [tuanda@master-node ]$ tree ├── api-depl.yaml ├── api-service.yaml └── kustomization.yaml # vim kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - api-depl.yaml - api-service.yaml # Gán transformer namespace cho toàn bộ .yaml namespace: tuanda # Đổi transformer image và tag trong deployment nếu trùng "name" images: - name: httpd newName: nginx newTag: 1.21.0 #transformer Đặt tên đầu và cuối cho .yaml. #VD: xxx thành LAB-xxx-dev namePrefix: LAB- nameSuffix: -dev #Thêm Annotations commonAnnotations: branch: master #Thêm Labels vào toàn bộ yaml commonLabels: someName: someValue owner: tuanda app: bingo |
Result:
1 2 | # kustomize build . |
P3: Overlays-base
Concept: To separate the parameters by environment ( prod/stag/dev ). We can use kustomize overlays.
In which the general parameters will be placed in the “base” directory, the environment-specific configurations (such as replicas, secret-configmap) will be placed in the “overlays” directory.
1. Create a basic directory tree
1 2 3 4 5 6 | # cd kustomize-sample # mkdir -p 03.overlays/overlays #(thư mục base đã được tạo ở ví dụ 1) # mkdir -p 03.overlays/overlays/prod # mkdir -p 03.overlays/overlays/stag # mkdir -p 03.overlays/overlays/dev |
2. Edit kustomization.yaml in all 3 subfolders of overlays (prod/stag/dev) with the following content
1 2 3 4 5 6 7 8 9 10 11 12 13 | # cp -rp kustomize-sample/02.transformer/base kustomize-sample/03.overlays/ #(mượn tạm thư mục base của phần 2, hoặc bạn tự tạo lại bằng kubecli) # vim kustomize-sample/03.overlays/overlays/prod/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization nameSuffix: -prod #(chú ý đổi Suffix trong cả 3 file) bases: - ../../base resources: - config-map.yaml #(mỗi 1 môi trường sẽ có config-map.yaml riêng) |
3. Create config-map.yaml for each environment
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 | # vim kustomize-sample/03.overlays/overlays/prod/config-map.yaml apiVersion: v1 kind: ConfigMap metadata: name: database-cfg data: password: prod-pwd-123 username: prod-acc-123 # vim kustomize-sample/03.overlays/overlays/stag/config-map.yaml apiVersion: v1 kind: ConfigMap metadata: name: database-cfg data: password: stag-pwd-123 username: stag-acc-123 # vim kustomize-sample/03.overlays/overlays/dev/config-map.yaml apiVersion: v1 kind: ConfigMap metadata: name: database-cfg data: password: dev-pwd-123 username: dev-acc-123 |
4. Run kustomize to check if the BASE declaration is correct
1 2 3 4 5 6 7 8 9 | # cd kustomize-sample/03.overlays/overlays/prod kustomize build . # cd kustomize-sample/03.overlays/overlays/stag kustomize build . # cd kustomize-sample/03.overlays/overlays/dev kustomize build . |
Result: Each environment we deploy api-depl.yaml and api-service.yaml, but config-map.yaml account and password will be different for each environment
P4: Patches
(Refer:)
Jsonpath6902: https://github.com/kubernetes-sigs/kustomize/blob/master/examples/jsonpatch.md
Strategic merge: https://fabianlee.org/2022/04/18/kubernetes-kustomize-transformations-with-patchesstrategicmerge/
Patches have 3 editing functions on yaml k8s: replace, delete, add
Example 1: Replace patch
The topic posed:
1 2 3 4 | - Prod tôi muốn sửa 10 pod cho nginx - Stag tôi muốn sửa 9 pod cho nginx - Dev tôi muốn sửa 8 pod cho nginx, thay image nginx:alpine thành httpd:alpine |
Solution: There are 2 ways to use Json6092 or Strategic Merge Patch
Solution 1: Json6902 patch
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 | # vim kustomize-sample/04.patches/vd1-replace/json6902/overlays/prod/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization nameSuffix: -prod #(chú ý đổi Suffix trong cả 3 file) bases: - ../../base resources: - config-map.yaml #patch json6902 demo replace patches: - target: kind: Deployment name: nginx-api patch: |- - op: replace path: /spec/replicas value: 10 -------------------------- # # vim kustomize-sample/04.patches/vd1-replace/json6902/overlays/stag/kustomization.yaml > tương tự như prod trên, sửa value 10 thành 9 -------------------------- # vim kustomize-sample/04.patches/vd1-replace/json6902/overlays/dev/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization nameSuffix: -dev bases: - ../../base resources: - config-map.yaml #demo patches json6902 replace dev patches: - target: kind: Deployment name: nginx-api patch: |- - op: replace path: /spec/replicas value: 8 - target: kind: Deployment name: nginx-api patch: |- - op: replace path: /spec/template/spec/containers/0 value: name: newnametest image: httpd:alpine |
Result:
> the results are in stag, dev run by yourself (kustomize build .).
Solution 2: Strategic merge patch
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 | # vim kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization nameSuffix: -prod #(chú ý đổi Suffix trong cả 3 file) bases: - ../../base resources: - config-map.yaml #patch strategic-merge demo replace patches: - patch: |- apiVersion: apps/v1 kind: Deployment metadata: creationTimestamp: null labels: app: nginx-api name: nginx-api spec: replicas: 10 (ta có 1 cách khác là load từ file path/api-depl.yaml, bạn có thể tham khảo thư mục stag trên git của tôi) -------------------------- # kustomize build . |
Result
You can apply kustomize patches to replace any parameters you want: for example in services switch the NodePort/ClusterIP/Balance type for each environment
Example 2: Add patch
1 2 3 4 5 6 7 8 9 10 11 12 13 14 | #VD2----JSON6902 add----# #demo patches json6902 add patches: #Patch add logstash service mesh - target: kind: Deployment name: nginx-api patch: |- - op: add path: /spec/template/spec/containers/- value: name: logstash-mesh image: logstash:latest |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 | #VD2----Strategic Merge add----# #patch strategic-merge demo add patches: - patch/api-depl.yaml # vim patch/api-depl.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-api spec: template: spec: containers: - image: nginx:alpine name: nginx - image: logstash:latest name: logstash-mesh |
The same result both ways:
Example 3: Delete patch
1 2 3 4 5 6 7 8 9 10 11 12 | #----JSON6902 remove----# #demo patches json6902 delete creationTimestamp patches: - target: kind: Deployment name: nginx-api patch: |- - op: remove path: /spec/template/metadata value: - creationTimestamp |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 | #----Strategic Merge remove----# #patch strategic-merge demo replace patches: - patch/api-depl.yaml # vim patch/api-depl.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx-api spec: template: spec: containers: - $patch: delete name: tomcat |
P6: Component
(Reference: https://github.com/kubernetes-sigs/kustomize/blob/master/examples/components.md )
The purpose of the generated Component is to reuse the components that need to be reused for each environment. For example, we have the following table
| Overlays | External-DB history | Cache |
|---|---|---|
| prod | Yes | Yes |
| stag | No | Yes |
| dev | Options | Options |
As the table above.
- Prod needs extra DB to store user history data, and cache to increase data load from ram
- Stag doesn’t need to DB heavy traffic of historical data, need cache to test performance
- Dev can add more DB and Cache if needed to test, or not. We will design the component file as follows:
Edit the kustomization.yaml file according to each environment. (Yaml sample: https://github.com/worldhello12/kustomize-sample/tree/main/05.component )
1 2 3 4 5 6 7 8 9 10 11 | # vim prod/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization bases: - ../../base components: - ../../components/external_db - ../../components/cache |
File components/external_db/kustomization.yaml. We notice kind, apiVersion will be Component
1 2 3 4 5 6 7 8 | # vi components/external_db/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1alpha1 kind: Component resources: - external_db_cfg.yaml - external_db_depl.yaml |
Run kustomize build on all 3 environments, we will have the same results as the table.
| Overlays | External-DB history | Cache |
|---|---|---|
| prod | Yes | Yes |
| stag | No | Yes |
| dev | Options | Options |
.
The article is within the knowledge of the editor, if you have a better solution, please recommend and comment. thank.