[K8S] Part 11 – Building a Private Docker Registry for CICD with Kubernetes

Tram Ho

# Foreword Hi guys, continuing with the kubernetes lab building series, deploying applications on K8S will necessarily use **Docker Registry** to store Docker Image for the project. Depending on the characteristics of each company, different solutions are used. For systems on the cloud, it is possible to use the available Docker Registry services of Cloud Vendors. But with Onpremis, you will have to build your own. I see there are many Docker Registry solutions such as using Gitlab Registry (which comes with Gitlab suite is quite convenient), Private Docker Registry, Harbor Registry .. In this article, I will guide you to build Docker Registry using **Private Docker Registry ** and **Harbor Registry** installed on Centos. Depending on your needs, which one is more suitable, you can refer to it and use it. ***Note: I save part 9 to guide the installation of some popular open-source software and part 10 to install logging for K8S. However, today it’s convenient to install the registry, so write this article first :”>*** # Introduction Usually when you have a client computer that has docker installed, you can perform operations on it with docker like docker pull to download docker image When executing the command: docker pull :version Then the docker client will make a connection to the default docker hub to find and pull that docker image to the client machine. Docker Hub is Docker’s service for finding and sharing Docker images for everyone. However, when you deploy a project and require your own Docker Registry, or because conditions do not allow an external connection to the Internet to download from Docker Hub. Then you will think about needing to build a separate Docker Registry. Private Docker Registry will help you manage Docker images and share them with everyone on the team. It also makes sharing/downloading Docker images for local connection only, no internet connection required. This also helps to reduce deployment time because the bandwidth of the local network can be much larger than that of the Internet. Also, companies will not save images on public platforms, but will have to have their own management system. ![image.png](https://images.viblo.asia/86fd84a1-dbba-4aef-a706-ae8a7b1f0a2c.png) ***Information about the lab I’m building:*** – 03 Master Node: viettq- master1 (192.168.10.11), viettq-master2 (192.168.10.12), viettq-master3 (192.168.10.13) – 03 Worker Node: viettq-workter1 (192.168.10.14), viettq-workter2 (192.168.10.15), viettq-workter3 (192.168.10.16) – 01 Rancher Node: viettq-rancher (192.168.10.19) – 01 CICD Node: viettq-cdcd (192.168.10.20) ***Due to the purpose of building Kubernetes + CICD lab, I built Registry on CICD node , for future use.*** # Instructions to install Harbor Registry I will install Harbor Registry and use the domain name ***harbor.prod.viettq.com***. Here I will reuse the previously generated cert (used to configure SSL Terminate in Haproxy) so that I don’t have to re-generate it. You can refer to the cert creation here: ***https://viblo.asia/p/k8s-phan-7-huong-dan-tao-tich-xanh-cho-ung-dung-tren-k8s -dung-openssl-djeZ1EP8ZWz*** ## Create installation directory and data directory I create a directory to save installation configurations at **/home/sysadmin/open-sources/harbor_registry** and create my own registry data directory at **/data/harbor_data**. Note that I will use the root user to install. bash sudo -s mkdir -p /data/harbor_data mkdir -p /home/sysadmin/open-sources/harbor_registry cd /home/sysadmin/open-sources/harbor_registry curl -s https://api.github.com /repos/goharbor/harbor/releases/latest | grep browser_download_url | cut -d '"' -f 4 | grep '.tgz$' | wget -i - tar xvzf harbor-offline-installer*.tgz cd harbor cp harbor.yml.tmpl harbor.yml ## Prepare certificate for registry Create a directory containing the Cert: bash mkdir -p /home/sysadmin/open-sources/harbor_registry/certs cd /home/sysadmin/open-sources/harbor_registry/certs vi harbor.prod.viettq.com. key vi harbor.prod.viettq.com.crt 2 files *** .key** and ***.crt** above I took from the file created according to the instructions in part 7! Reference: ***https://viblo.asia/p/k8s-phan-7-huong-dan-tao-tich-xanh-cho-ung-dung-tren-k8s-dung-openssl-djeZ1EP8ZWz*** Result: bash [sysadmin@vtq-cicd certs]$ ls -lrt total 8 -rw-rw-r-- 1 sysadmin sysadmin 1679 Apr 23 06:10 harbor.prod.viettq.com.key -rw-rw -r-- 1 sysadmin sysadmin 1354 Apr 23 06:10 harbor.prod.viettq.com.crt ## Configure the harbor.yaml installation file Go back to the installation directory to edit the installation file from the Harbor template : bash cd /home/sysadmin/open-sources/harbor_registry cd harbor vi harbor.yaml Friends need to update some parameters as follows: yaml hostname: harbor.prod.viettq.com certificate: /home/sysadmin/open-sources/harbor_registry/certs/harbor.prod.viettq.com.crt private_key: /home /sysadmin/open-sources/harbor_registry/certs/harbor.prod.viettq.com.key harbor_admin_password: Harbor_123 password: root_123 data_volume: /data/harbor_data Make sure User root can run the docker-compose command. Because I installed docker-compose with the sysadmin user, the root user can’t see docker-compose in its path, so I have to do the following further: bash [sysadmin@vtq-cicd harbor]$ sudo -s [root@vtq -cicd harbor]# which docker-compose /usr/bin/which: no docker-compose in (/sbin:/bin:/usr/sbin:/usr/bin) [root@vtq-cicd harbor]# exit [sysadmin @vtq-cicd harbor]$ which docker-compose /usr/local/bin/docker-compose [sysadmin@vtq-cicd harbor]$ sudo -s [root@vtq-cicd harbor]# cp /usr/local/bin/ docker-compose /usr/bin/ ## Install harbor registry After configuration is complete, install with 1 command: bash cd /home/sysadmin/open-sources/harbor_registry/harbor . /install.sh The result when running successfully will have the following containers: bash [root@vtq-cicd harbor]# docker-compose ps Name Command State Ports --------- --------------------------------------------------- --------------------------------------------------- ----------------------------------- harbor-core /harbor/entrypoint.sh Up (health y) harbor-db /docker-entrypoint.sh 96 13 Up (healthy) harbor-jobservice /harbor/entrypoint.sh Up (healthy) harbor-log /bin/sh -c /usr/local/bin/ ... Up (healthy) 127.0.0.1:1514->10514/tcp harbor-portal nginx -g daemon off; Up (healthy) nginx nginx -g daemon off; Up (healthy) 0.0.0.0:80->8080/tcp,:::80->8080/tcp, 0.0.0.0:443->8443/tcp,:::443->8443/tcp redis redis-server / etc/redis.conf Up (healthy) registry /home/harbor/entrypoint.sh Up (healthy) registryctl /home/harbor/start.sh Up (healthy) ## Harbor Web-UI connection You declare host on the client as follows: 192.168.10.20 harbor.prod.viettq.com Then go to ***https://harbor.prod.viettq.com***. The reason I declare host and connect by domain is to use SSL configuration: ![image.png](https://images.viblo.asia/a66636a6-915c-432a-b0d6-47c785b67202.png) Login with admin user, pass as set in the configuration file above. ### Configure the project and assign user permissions Go to Users -> NEW USER and then enter information to create a user. I created a user/pass as **demo/Demo_123** Go to Projects to create a project and assign permissions to the newly created user: ![image.png](https://images.viblo.asia/4f989247-a463-4fa0-8706 -61279dbba17d.png) Assign permission to user **demo** to this project: Go to project **demo**, select Members –> +USER: ![image.png](https://images.viblo.asia /1f617611-dc38-49ad-baa9-db0b240a1ffc.png) ## Configure client connection (to pull/push image) Declare host for client: 192.168.10.20 harbor.prod.viettq.com Create directory containing cert on client: bash sudo -s mkdir -p /etc/docker/certs.d/harbor.prod.viettq.com/ cd /etc/docker/certs.d/harbor.prod.viettq .com/ vi harbor.prod.viettq.com.crt In which the content of the file **harbor.prod.viettq.com.crt** is the file I created to install harbor as in the above step. Tag an image for this registry and push it up. I will pull the **hello-world** image from github to tag and push to this Harbor Registry: [sysadmin@vtq-cicd harbor]$ docker pull hello-world Using default tag: latest latest: Pulling from library/ hello-world Digest: sha256:10d7d58d5ebd2a652f4d93fdd86da8f265f5318c6a73cc5b6a9798ff6d2b2e67 Status: Image is up to date for hello-world:latest docker.io/library/hello-world:latest [sysadmin@vt]: docker harbor hello harbor] prod.viettq.com/demo/hello-world:latest [sysadmin@vtq-cicd harbor]$ docker login harbor.prod.viettq.com Username: demo Password: WARNING! Your password will be stored unencrypted in /home/sysadmin/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded [sysadmin@vtq-cicd harbor]$ docker push harbor.prod.viettq.com/demo/hello-world: latest The push to repository [harbor.prod.viettq.com/demo/hello-world] e07ee1baac5f: Pushed latest: digest: sha256:f54a58bc1aac5ea1a25d796ae155dc228b3f0e11d046ae276b39c4bf2f13d8c4 size *** registry already. Note before pushing, you must execute the docker login command and enter the user information of the harbor as I created before!*** # This guide to install Private Docker Registry Registry is simple and lightweight. And with that, it also has very few features, does not support decentralization.. In general, it does the right job of managing Docker Image, enough for you to use to pull / push images. ## Create installation directory Requires the machine to install Docker and docker-compose. At this point, this installation is mandatory for you to know, so I will not introduce it here anymore. I created a folder to save the installation file and the data folder in the /data partition is a separate partition to save data (in prod environments, the data storage partition will usually run from SAN first): bash mkdir -p /home/sysadmin/open-sources/private-docker-registry mkdir -p /home/sysadmin/open-sources/private-docker-registry/certs cd /home/sysadmin/open-sources/private-docker -registry sudo mkdir /data/registry_data sudo chown -r sysadmin:sysadmin /data/registry_data ## Create a certificate for the Private Docker Registry You can refer to this Certificate section in my previous post here: ***https://viblo.asia/p/k8s-phan-7-huong-dan-tao-tich-xanh-cho-ung-dung-tren-k8s-dung-openssl-djeZ1EP8ZWz*** In this section We will create a CA, then we will create an SSL Cert for the Registry signed by the CA created. ### Generate Certificate Authority Execute the following command to generate a Private Key for the CA, enter the password when prompted and remember this password: bash cd /home/sysadmin/open-sources/private-docker-registry/certs openssl genrsa -des3 -out rootCA.key 2048 Enter pass phrase for rootCA.key: Verifying – Enter pass phrase for rootCA.key: Generate pem file from private key file (enter password of rootCA created above): bash [root@vtq-cicd]# openssl req -x509 -new -nodes -key rootCA.key -sha256 -days 1825 -out rootCA.pem Enter pass phrase for rootCA.key: You are about to be asked to enter information that will be incorporated into your certificate request. What you are about to enter is what is called a Distinguished Name or a DN. There are quite a few fields but you can leave some blank For some fields there will be a default value, If you enter ‘.’, the field will be left blank. —– Country Name (2 letter code) [XX]:VN State or Province Name (full name) []:HN Locality Name (eg, city) [Default City]:HN Organization Name (eg, company) [ Default Company Ltd]:VietTQ_CA Organizational Unit Name (eg, section) []:VietTQ_CA_Unit Common Name (eg, your name or your server’s hostname) []:private-docker-registry.viettq.com Email Address []:viettq@email .com After this step you will see 2 files generated: bash [sysadmin@vtq-cicd certs]$ ls -lrt total 8 -rw-rw-r– 1 sysadmin sysadmin 1751 Apr 23 03 :36 rootCA.key -rw-rw-r– 1 sysadmin sysadmin 1460 Apr 23 03:39 rootCA.pem ### Generate SSL Certificate for Private Docker Registry First we create a file openssl.cnf to configure add the SAN information as follows [req] distinguished_name = req_distinguished_name req_extensions = v3_req [req_distinguished_name] countryName = VN countryName_default = VN stateOrProvinceName = HN stateOrProvinceName_default = HN localityName = HN localityName_default = HN organizational tTQ_DEVOPS organizationalUnitName_default = VietTQ_DEVOPS commonName = private-docker-registry.viettq.com commonName_max = 64 [ v3_req ] # Extensions to add to a certificate request basicConstraints = CA:FALSE keyUsage = nonRepudiation, digitalSignature, keyEncipherment subject_names] 1 = *.viettq.com Here I create SSL Certcificate for the domain of Registry is *.viettq.com. Next, create a key file for the domain: bash sudo openssl genrsa -out private-docker-registry.key 2048 Then we create a file Sigining Request from the key file and the above config file: bash sudo openssl req -new -out private-docker-registry.csr -key private-docker-registry.key -config openssl.cnf The result is a **private-docker-registry.csr** file. This CSR file is like an application for approval, a certificate that I am the domain **private-docker-registry.viettq.com**. The person who approves is the CA (Certificate Authority) to create a certificate file with the extension .crt. [sysadmin@vtq-cicd certs]$ sudo openssl x509 -req -days 3650 -in private-docker-registry.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -out private-docker-registry.crt -extensions v3_req -extfile openssl.cnf Signature ok subject=/C=VN/ST=HN/L=HN/OU=VietTQ_DEVOPS/CN=private-docker-registry.viettq.com Getting CA Private Key Enter pass phrase for rootCA. key: [sysadmin@vtq-cicd certs]$ ls -lrt total 28 -rw-rw-r– 1 sysadmin sysadmin 1751 Apr 23 03:36 rootCA.key -rw-rw-r– 1 sysadmin sysadmin 1460 Apr 23 03:39 rootCA.pem -rw-r–r– 1 root root 1675 Apr 23 03:49 private-docker-registry.key -rw-rw-r– 1 sysadmin sysadmin 581 Apr 23 03:52 openssl. cnf -rw-r–r– 1 root root 1098 Apr 23 03:52 private-docker-registry.csr -rw-r–r– 1 root root 17 Apr 23 03:57 rootCA.srl -rw- r–r– 1 root root 1359 Apr 23 03:57 private-docker-registry.crt As a result we have 2 files needed to install are **private-docker-registry.key** and * *private-docker-registry.crt** ### Install Private Docker Regist ry with docker-compose Create a docker-compose.yaml file with the following content: yaml version: ‘3’ services: docker-registry: image: registry:2 container_name: private-docker-registry ports: – 5000: 5000 restart: always environment: REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /data REGISTRY_HTTP_TLS_CERTIFICATE: /certs/private-docker-registry.crt REGISTRY_HTTP_TLS_KEY: /certs/private-docker-registry.key REGISTRY_STORDAGE_CACHE : registry_data:/data – /home/sysadmin/open-sources/private-docker-registry/certs:/certs links: – redis-registry docker-registry-ui: image: konradkleine/docker-registry-frontend:v2 container_name: docker -registry-ui restart: always ports: – 8088:80 environment: ENV_DOCKER_REGISTRY_USE_SSL: 1 ENV_DOCKER_REGISTRY_HOST: private-docker-registry ENV_DOCKER_REGISTRY_PORT: 5000 redis-registry: image: redis:latest container_name: redis 6379 Next is installation n with the command docker-compose: bash [sysadmin@vtq-cicd private-docker-registry]$ docker-compose up -d Creating network “private-docker-registry_default” with the default driver Creating docker-registry-ui . .. done Creating redis-registry … done Creating private-docker-registry … done [sysadmin@vtq-cicd private-docker-registry]$ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 71987c2855e5 registry:2 “/ entrypoint.sh /e…” 3 seconds ago Up 3 seconds 0.0.0.0:5000->5000/tcp private-docker-registry f3899ed5d349 docker.io/redis:latest “docker-entrypoint…” 4 seconds ago Up 3 seconds 0.0.0.0:6379->6379/tcp redis-registry 7ab9c04381fa konradkleine/docker-registry-frontend:v2 “/bin/sh -c $START…” 4 seconds ago Up 3 seconds 443/tcp, 0.0. 0.0:8088->80/tcp docker-registry-ui ### Client configuration With client nodes to be able to pull/pull images from this registry, we need to add the following declaration step: bash sudo -s mkdir -p /etc/docker/certs.d/private-docker-registry.v iettq.com:5000 cd /etc/docker/certs.d/private-docker-registry.viettq.com:5000/ vi ca.crt You need to fill in the contents of the ca.crt file with the file ** private-docker-registry.crt** created earlier. Next is to declare the host file for the client: 192.168.10.20 private-docker-registry.viettq.com Where 192.168.10.20 is the IP of the node where the Private Docker Registry is installed. Now you can try to tag the image and push it to the registry: bash [sysadmin@vtq-cicd private-docker-registry]$ docker tag hello-world:latest private-docker-registry.viettq.com:5000/ demo/hello-world:latest [sysadmin@vtq-cicd private-docker-registry]$ docker push private-docker-registry.viettq.com:5000/demo/hello-world:latest The push refers to a repository [private- docker-registry.viettq.com:5000/demo/hello-world] e07ee1baac5f: Pushed latest: digest: sha256:f54a58bc1aac5ea1a25d796ae155dc228b3f0e11d046ae276b39c4bf2f13d8c4 ### View registry information on Web-UI You also declare the host for the client to access web-io with the domain name. 192.168.10.20 private-docker-registry.viettq.com Then go to the browser to connect to the address: ***private-docker-registry.viettq.com:8088*** ![image .png](https://images.viblo.asia/69f2db23-517b-4ef9-82b5-445185d5beda.png) Go inside the repositories to see the image I pushed: ![image.png](https://images. viblo.asia/5ac5fd72-1990-4b62-9fd8-1266cf317bd6.png) ***Thanks everyone for watching and see you in the next parts!***

Share the news now

Source : Viblo