基于 Kubernetes 的 CI/CD

模拟实际生产环境部署离线 Kubernetes 和 CI/CD 环境，并测试

周五 12月 27 2024
4334 字 · 27 分钟
Documentation 云运维工坊 Kubernetes Harbor CI/CD DevOps Jenkins git

容器编排技术

一、前言

  1. 了解Jenkins的离线安装步骤
  2. 掌握Gitlab的使用和管理
  3. 了解CICD的配置步骤和方法

二、基础环境部署

机器IP作用
master192.168.100.10K8s 的 master 节点、Harbor 节点、CI/CD 节点
node192.168.100.20K8s 的 node 节点

​ 在这两台机器上提前部署好 CentOS 7 + Docker + Kubernetes12 + Harbor3 环境，准备好 XianDian-PaaS-v2.4.iso 包45

1. 安装 Jenkins 环境

1.1 查看 K8s 集群状态和节点信息
BASH
$ kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE   ERROR
scheduler            Healthy   ok        
controller-manager   Healthy   ok        
etcd-0               Healthy
$ kubectl get nodes
NAME     STATUS   ROLES           AGE   VERSION
master   Ready    control-plane   31d   v1.27.16
node     Ready    <none>          31d   v1.27.16
1.2 安装 Jenkins
BASH
$  docker run -d --name jenkins -p 8080:8080 -u root \
-v /home/jenkins_home:/var/jenkins_home \
-v /var/run/docker.sock:/var/run/docker.sock \
-v $(which docker):/usr/bin/docker \
-v /usr/bin/kubectl:/usr/local/bin/kubectl \
-v /root/.kube:/root/.kube \
jenkins/jenkins:latest-jdk21
22f894f46083c2b8836ae4e4c67e0bf4e8cd2e09adcc29417d6b8b473dea1407

​ 然后在电脑浏览器输入 http://192.168.100.10:8080 来访问 Jenkins

2.1.2.1

​ 然后进入容器，输入密码

BASH
$ docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword
2169d1cbaa1546f396c0571477d2ce6a

​ 下面的 2169d1cbaa1546f396c0571477d2ce6a 就是我们的密码，输入密码并点击 “继续”

2.1.2.2

​ 然后点击 “安装推荐的插件”

2.1.2.3

​ 插件安装完成后自动转入管理员用户创建界面

2.1.2.4

​ 输入用户信息，点击“保存并完成”

​ 配置完URL后使用新建的用户登录 Jenkins

2.1.2.5

2. 部署 GitLab

2.1 启动 GitLab

​ GitLab 是利用 Ruby on Rails 一个开源的版本管理系统，实现一个自托管的 Git 项目仓库，可通过 Web 界面进行访问公开的或者私人项目。

​ 与 Github 类似，GitLab 能够浏览源代码，管理缺陷和注释，可以管理团队对仓库的访问，它非常易于浏览提交过的版本并提供一个文件历史库，团队成员可以利用内置的简单 聊天程序(Wall) 进行交流。Gitlab 还提供一个代码片段收集功能可以轻松实现代码复用，便于日后有需要的时候进行查找

​ 本项目Gitlab与Harbor共用一台服务器

BASH
$ docker run -d -h gitlab -p 1022:22 -p 81:80 -p 443:443 \
--volume /srv/gitlab/config:/etc/gitlab \
--volume /srv/gitlab/gitlab/logs:/var/log/gitlab \
--volume /srv/gitlab/gitlab/data:/var/opt/gitlab \
--restart always --name mygitlab gitlab/gitlab-ce:latest
e87a1e5298f2c0430ad5f2269472e9f06cc2eddf71477fb28a6e8089cf46f8d4

​ GitLab 启动较慢，可以通过 docker logs 查看启动状态

BASH
$ docker logs mygitlab

​ 启动完成后，在浏览器上访问 http://192.168.100.10:81 来进入刚刚搭建的本地 GitLab

​ 然后设置 root 用户信息

2.2.1.1

​ 然后使用 root 用户登录

2.2.1.22.2.1.3

2.2 创建项目

​ 点击 “Create a project”(创建项目)，创建项目 “ChinaskillProject”，Visibility Level(可见等级) 设置为 “Public”

2.2.2.1

​ 点击 “Create Project”(创建项目)，然后自动进入项目

2.2.2.2

​ 根据提示，配置 SSH 公钥到 GitLab 上

BASH
$ ssh-keygen -t rsa
$ cat ~/.ssh/id_rsa.pub

​ 然后复制出来的内容，到 GitLab 上

​ GitLab 上点击 “Add SSH Key”(添加 SSH 密钥)，在 Key 栏里面粘贴你刚才复制的内容，Title 默认即可，Expires at(有效期截至到) 随便选，然后点击 “Add Key”(添加密钥)

2.2.2.3

​ 然后将 之前准备好的 XianDian-PaaS-v2.4.iso4 上传并挂载到虚拟机， git push 源代码到 GitLab 的 ChinaskillProject 项目

BASH
$ sudo yum install -y git
$ mount /root/XianDian-PaaS-v2.4.iso /mnt/
$ cp -rf /mnt/ChinaskillProject/ /opt
$ ls -al /opt/ChinaskillProject/
总用量 36
dr-xr-xr-x  13 root root 4096 12月 25 11:18 .
drwxr-xr-x.  5 root root   60 12月 25 11:18 ..
dr-xr-xr-x   3 root root   50 12月 25 11:18 account-service
dr-xr-xr-x   3 root root   50 12月 25 11:18 auth-service
dr-xr-xr-x   3 root root   50 12月 25 11:18 config
-r-xr-xr-x   1 root root  823 12月 25 11:18 docker-compose.dev.yml
-r-xr-xr-x   1 root root 4020 12月 25 11:18 docker-compose.yml
dr-xr-xr-x   3 root root   50 12月 25 11:18 gateway
dr-xr-xr-x   8 root root  185 12月 25 11:18 .git
-r-xr-xr-x   1 root root   68 12月 25 11:18 .gitignore
-r-xr-xr-x   1 root root 1107 12月 25 11:18 LICENCE
dr-xr-xr-x   3 root root   51 12月 25 11:18 mongodb
dr-xr-xr-x   3 root root   50 12月 25 11:18 monitoring
dr-xr-xr-x   3 root root   50 12月 25 11:18 notification-service
-r-xr-xr-x   1 root root  723 12月 25 11:18 pom.xml
-r-xr-xr-x   1 root root 1075 12月 25 11:18 README.md
dr-xr-xr-x   3 root root   50 12月 25 11:18 registry
dr-xr-xr-x   3 root root   50 12月 25 11:18 statistics-service
-r-xr-xr-x   1 root root 4428 12月 25 11:18 .travis.yml
dr-xr-xr-x   4 root root   35 12月 25 11:18 yaml

$ cd /opt/ChinaskillProject/
# 移除“指定远程目录”
$ git remote remove origin
# 指定“我们自己的”远程目录
$ git remote add origin http://192.168.100.10:81/root/chinaskillproject.git

$ git add .
$ git commit -m "Initial commit"
$ git push -u origin master
# 这里输入 root
Username for 'http://192.168.100.10:81': root
# 这里设置你刚才配置 GitLab 的时候配置的密码
Password for 'http://root@192.168.100.10:81': 
Counting objects: 3192, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (1428/1428), done.
Writing objects: 100% (3192/3192), 1.40 MiB | 0 bytes/s, done.
Total 3192 (delta 1233), reused 3010 (delta 1207)
remote: Resolving deltas: 100% (1233/1233), done.
To http://192.168.100.10:81/root/chinaskillproject.git
 * [new branch]      master -> master
分支 master 设置为跟踪来自 origin 的远程分支 master。

​ 刷新网页，ChinaskillProject 项目中已经有我们刚刚 git push 上去的文件了

2.2.2.4

3. 配置 Jenkins 连接 GitLab

3.1 设置 Outbound requests(出站请求)

​ 登录 GitLab，打开 Dashboard，然后点击 🔧(settings)

​ 然后点击 “Settings” → “Network” → “Outbound requests”(出战请求) 旁边的 Expand(展开)

​ 然后勾选 “Allow requests to the local network from web hooks and services”

​ 在下面的输入栏里面输入 192.168.100.0/24

​ 然后点击 “Save change”(保存设置)

2.3.1.1

3.2 创建 Github API Token

​ 点击 GitLab 的用户头像，然后点击 “Settings” → “Access Token” 添加一个 Token

​ 在 Scopes(范围) 上选上 “api” 和 “read_user”，其他的和上面一样，随便填一个就行，然后点击 “Create personal access token”(创建个人访问令牌)

2.3.2.1

​ 然后复制 “Your New Personal Access Token”(你的新个人访问令牌) 栏里面的内容，后面配置 Jenkins 的时候需要用到

TEXT
bWxAexndVn6N-XsmFHWz

2.3.2.2

3.3 设置 Jenkins

​ 登录 Jenkins Dashboard，然后点击 “Manage Jenkins”(管理 Jenkins/系统管理) → “Plugins”(插件) → “Installed plugins”(已安装插件) 查看有没有以下插件

  • “GitLab”
  • “Generic Webhook Trigger”
  • “GitLab API”
  • “GitLab Authentication”
  • “GitLab Branch Source”
  • “Gitlab Merge Request Builder”
  • “GitLab Logo”

​ 如果没有的话就到 “Available plugins”(可用插件) 去安装一下

​ 然后回到 Jenkins Dashboard，点击 “Manage Jenkins”(管理 Jenkins/系统管理) → “System”(系统配置)

​ 然后配置 GitLab 信息，取消勾选 “Enable authentication for ‘/project’ end-point” 然后在 “Credentials” 下面点击 “添加”

​ 然后在 类型 里面选择 “GitLab API token”，“API token” 填入刚刚复制的 token

​ 然后点击 “Test Connection”(测试连接)

2.3.3.12.3.3.2

4. 配置 Jenkins 连接 maven

4.1 安装 maven

​ 由于我们这里的 Jenkins 是采用 Docker in Docker 的方式启动的，所以需要在 Jenkins 容器内安装 maven

BASH
# 下载 maven-3.6.3
$ curl -o /home/jenkins_home/apache-maven-3.6.3-bin.tar.gz https://archive.apache.org/dist/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz

# 进入 Docker 容器
$ docker exec -it jenkins /bin/bash
	$ tar -zxvf /var/jenkins_home/apache-maven-3.6.3-bin.tar.gz -C .
	$ mv apache-maven-3.6.3/ /usr/local/maven
	$ apt update -y && apt upgrade -y
	$ apt install -y vim
	$ vim ~/.bashrc
	# 文末加入
	export M2_HOME=/usr/local/maven
	export PATH=$PATH:$M2_HOME/bin
	
	$ source ~/.bashrc
	$ exit
# 退出容器重新进入
$ docker exec -it jenkins /bin/bash
	$ mvn -v
	Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
	Maven home: /usr/local/maven
	Java version: 21.0.5, vendor: Eclipse Adoptium, runtime: /opt/java/openjdk
	Default locale: en, platform encoding: UTF-8
	OS name: "linux", version: "3.10.0-1160.71.1.el7.x86_64", arch: "amd64", family: "unix"
4.2 连接 maven

​ 登录 Jenkins，点击 “Manage Jenkins”(管理 Jenkins/系统管理) → “Tools”(工具/全局系统配置)

​ 往下翻找到 Maven，点击“新增Maven”，此处不要开启自动安装，所以不要勾选“自动安装”

​ 然后把填入 Maven 路径填入 MAVEN_HOME ，应用保存即可

2.4.2.1

5. 部署 Harbor 仓库

5.1 安装 Harbor 仓库

​ Harbor 依赖 Docker 和 Docker Compose。因此，首先需要在系统中安装 Docker Compose

BASH
$ DOCKER_CONFIG=/usr/local/lib/docker/cli-plugins
$ sudo mkdir -p $DOCKER_CONFIG/cli-plugins
$ sudo curl -SL https://github.com/docker/compose/releases/download/v2.29.1/docker-compose-linux-x86_64 -o $DOCKER_CONFIG/cli-plugins/docker-compose
$ sudo chmod +x $DOCKER_CONFIG/cli-plugins
$ docker compose version

​ 然后下载 Offline 版的 Harbor

BASH
$ sudo curl -O https://github.com/goharbor/harbor/releases/download/v2.12.1/harbor-offline-installer-v2.12.1.tgz

# 国内用户可以使用以下方式加快下载
$ sudo curl -O https://moka.anitsuri.top//image/13/Registry+Harbor/harbor-offline-installer-v2.11.1.tgz

$ sudo tar -zxf harbor-offline-installer-v2.12.1.tgz
$ cd harbor

​ 然后修改配置文件，把 harbor.yml.tmpl 复制一个变成 harbor.yml

BASH
$ sudo cp harbor.yml.tmpl harbor.yml
$ sudo vi harbor.yml

​ 修改/填写以下内容

YAML
# Configuration file of Harbor

# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: 192.168.100.10

# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 80

# https related config
# https:
  # https port for harbor, default is 443
  # port: 443
  # The path of cert and key files for nginx
  # certificate: /your/certificate/path
  # private_key: /your/private/key/path
  # enable strong ssl ciphers (default: false)
  # strong_ssl_ciphers: false

# # Harbor will set ipv4 enabled only by default if this block is not configured
# # Otherwise, please uncomment this block to configure your own ip_family stacks
# ip_family:
#   # ipv6Enabled set to true if ipv6 is enabled in docker network, currently it affected the nginx related component
#   ipv6:
#     enabled: false
#   # ipv4Enabled set to true by default, currently it affected the nginx related component
#   ipv4:
#     enabled: true

# # Uncomment following will enable tls communication between all harbor components
# internal_tls:
#   # set enabled to true means internal tls is enabled
#   enabled: true
#   # put your cert and key files on dir
#   dir: /etc/harbor/tls/internal


# Uncomment external_url if you want to enable external proxy
# And when it enabled the hostname will no longer used
# external_url: http://192.168.92.128:8433

# The initial password of Harbor admin
# It only works in first time to install harbor
# Remember Change the admin password from UI after launching Harbor.
harbor_admin_password: Harbor12345

​ 在运行安装脚本前，需要先运行 prepare 脚本，来确保一切依赖和配置都已经就绪

BASH
$ ./prepare
prepare base dir is set to /root/harbor
Unable to find image 'goharbor/prepare:v2.12.1' locally
v2.12.1: Pulling from goharbor/prepare
0e5c20819ebf: Pull complete 
1fb69a2a4ee3: Pull complete 
c2aef12efe8a: Pull complete 
79fdf0882382: Pull complete 
eee2f1d222f1: Pull complete 
125a453a0ce4: Pull complete 
40b43804af0b: Pull complete 
5979e246bb31: Pull complete 
8d2c370a2c28: Pull complete 
53eaa20ee0df: Pull complete 
Digest: sha256:d0fcec61328185ddb5bfded6bf8b312428aa63b9bffe2a8d846452253234a898
Status: Downloaded newer image for goharbor/prepare:v2.12.1
WARNING:root:WARNING: HTTP protocol is insecure. Harbor will deprecate http protocol in the future. Please make sure to upgrade to https
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
Generated and saved secret to file: /data/secret/keys/secretkey
Successfully called func: create_root_cert
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir

$ ./install.sh

[Step 0]: checking if docker is installed ...

Note: docker version: 26.1.4

[Step 1]: checking docker-compose is installed ...

Note: Docker Compose version v2.27.1

[Step 2]: loading Harbor images ...
581136816168: Loading layer [==================================================>]  17.21MB/17.21MB
cbf4f3cb0c26: Loading layer [==================================================>]  3.584kB/3.584kB
19ba001b8b03: Loading layer [==================================================>]   2.56kB/2.56kB
b41af00dcd76: Loading layer [==================================================>]  70.97MB/70.97MB
5b5f57643342: Loading layer [==================================================>]  5.632kB/5.632kB
5175984e7e7a: Loading layer [==================================================>]  126.5kB/126.5kB
c8faf279c61c: Loading layer [==================================================>]  201.7kB/201.7kB
3a0151c1eba9: Loading layer [==================================================>]  72.09MB/72.09MB
dbf86e000945: Loading layer [==================================================>]   2.56kB/2.56kB
Loaded image: goharbor/harbor-core:v2.12.1
7d2be923c4b3: Loading layer [==================================================>]  146.8MB/146.8MB
1b7e2d8bb9c9: Loading layer [==================================================>]  3.584kB/3.584kB
c433c2b34c5a: Loading layer [==================================================>]  3.072kB/3.072kB
902158d6f915: Loading layer [==================================================>]   2.56kB/2.56kB
e795d71404bd: Loading layer [==================================================>]  3.072kB/3.072kB
cec684dd57a5: Loading layer [==================================================>]  3.584kB/3.584kB
47d0e4a2c5c7: Loading layer [==================================================>]  20.48kB/20.48kB
Loaded image: goharbor/harbor-log:v2.12.1
594f33e86793: Loading layer [==================================================>]  22.04MB/22.04MB
e0a4bafe73dc: Loading layer [==================================================>]  191.7MB/191.7MB
26b746af8db7: Loading layer [==================================================>]  26.16MB/26.16MB
31dcafd9cb03: Loading layer [==================================================>]  18.54MB/18.54MB
abf08c52686b: Loading layer [==================================================>]   5.12kB/5.12kB
0007d565ec69: Loading layer [==================================================>]  6.144kB/6.144kB
6e863404971d: Loading layer [==================================================>]  3.072kB/3.072kB
94c8eba384cc: Loading layer [==================================================>]  2.048kB/2.048kB
1a5955f6795e: Loading layer [==================================================>]   2.56kB/2.56kB
8ce6cc317185: Loading layer [==================================================>]   7.68kB/7.68kB
Loaded image: goharbor/harbor-db:v2.12.1
71775c4c621c: Loading layer [==================================================>]  17.21MB/17.21MB
4003ab14e211: Loading layer [==================================================>]  3.584kB/3.584kB
beabc71b1223: Loading layer [==================================================>]   2.56kB/2.56kB
579b2f017459: Loading layer [==================================================>]   59.9MB/59.9MB
895f989740a0: Loading layer [==================================================>]  60.69MB/60.69MB
Loaded image: goharbor/harbor-jobservice:v2.12.1
9590068fe044: Loading layer [==================================================>]  14.26MB/14.26MB
0cb250c20318: Loading layer [==================================================>]  4.096kB/4.096kB
9dac56331be1: Loading layer [==================================================>]  18.14MB/18.14MB
5dc6f164dc78: Loading layer [==================================================>]  3.072kB/3.072kB
141d394a5d47: Loading layer [==================================================>]  37.13MB/37.13MB
2587acb702ad: Loading layer [==================================================>]  56.05MB/56.05MB
Loaded image: goharbor/harbor-registryctl:v2.12.1
dc1a7f40e575: Loading layer [==================================================>]  17.21MB/17.21MB
3f56a5b9b5d4: Loading layer [==================================================>]  37.15MB/37.15MB
d67a91463720: Loading layer [==================================================>]  4.608kB/4.608kB
cedc4b1b129a: Loading layer [==================================================>]  37.94MB/37.94MB
Loaded image: goharbor/harbor-exporter:v2.12.1
37959c9a5eac: Loading layer [==================================================>]  137.2MB/137.2MB
Loaded image: goharbor/nginx-photon:v2.12.1
d07b5c3ce0a9: Loading layer [==================================================>]  137.2MB/137.2MB
f60722775a24: Loading layer [==================================================>]  6.731MB/6.731MB
4c9a05314a30: Loading layer [==================================================>]  252.9kB/252.9kB
61258323c496: Loading layer [==================================================>]  1.497MB/1.497MB
Loaded image: goharbor/harbor-portal:v2.12.1
9a1a8734fa3c: Loading layer [==================================================>]  14.75MB/14.75MB
856558c960ff: Loading layer [==================================================>]  4.096kB/4.096kB
9bd63d8a60d9: Loading layer [==================================================>]  3.072kB/3.072kB
a89a7d8b1691: Loading layer [==================================================>]  133.9MB/133.9MB
008e693070f3: Loading layer [==================================================>]  15.55MB/15.55MB
cef35bccdc97: Loading layer [==================================================>]  150.3MB/150.3MB
Loaded image: goharbor/trivy-adapter-photon:v2.12.1
d17edfb68476: Loading layer [==================================================>]  22.06MB/22.06MB
a0f8af0fdb08: Loading layer [==================================================>]  126.6MB/126.6MB
27e661303e23: Loading layer [==================================================>]  3.072kB/3.072kB
3f0390c3788e: Loading layer [==================================================>]   59.9kB/59.9kB
f56e18460c59: Loading layer [==================================================>]  61.95kB/61.95kB
Loaded image: goharbor/redis-photon:v2.12.1
8b7f3238d8aa: Loading layer [==================================================>]  14.26MB/14.26MB
295711278366: Loading layer [==================================================>]  4.096kB/4.096kB
ddadeb03375d: Loading layer [==================================================>]  3.072kB/3.072kB
99bbf20549c8: Loading layer [==================================================>]  18.14MB/18.14MB
9237fc270c32: Loading layer [==================================================>]  18.93MB/18.93MB
Loaded image: goharbor/registry-photon:v2.12.1
Loaded image: goharbor/prepare:v2.12.1


[Step 3]: preparing environment ...

[Step 4]: preparing harbor configs ...
prepare base dir is set to /root/harbor
WARNING:root:WARNING: HTTP protocol is insecure. Harbor will deprecate http protocol in the future. Please make sure to upgrade to https
Clearing the configuration file: /config/portal/nginx.conf
Clearing the configuration file: /config/log/logrotate.conf
Clearing the configuration file: /config/log/rsyslog_docker.conf
Clearing the configuration file: /config/nginx/nginx.conf
Clearing the configuration file: /config/core/env
Clearing the configuration file: /config/core/app.conf
Clearing the configuration file: /config/registry/passwd
Clearing the configuration file: /config/registry/config.yml
Clearing the configuration file: /config/registryctl/env
Clearing the configuration file: /config/registryctl/config.yml
Clearing the configuration file: /config/db/env
Clearing the configuration file: /config/jobservice/env
Clearing the configuration file: /config/jobservice/config.yml
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
loaded secret from file: /data/secret/keys/secretkey
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir


Note: stopping existing Harbor instance ...


[Step 5]: starting Harbor ...
[+] Running 10/10
 ✔ Network harbor_harbor        Created                                                                                    0.3s 
 ✔ Container harbor-log         Started                                                                                    2.0s 
 ✔ Container registryctl        Started                                                                                    2.9s 
 ✔ Container redis              Started                                                                                    2.9s 
 ✔ Container harbor-db          Started                                                                                    3.0s 
 ✔ Container harbor-portal      Started                                                                                    3.0s 
 ✔ Container registry           Started                                                                                    3.0s 
 ✔ Container harbor-core        Started                                                                                    3.2s 
 ✔ Container nginx              Started                                                                                    3.8s 
 ✔ Container harbor-jobservice  Started                                                                                    3.7s 
✔ ----Harbor has been installed and started successfully.----
5.2 配置 Harbor 到 Docker Daemon 文件

​ 在两台机器上，编辑 /etc/docker/daemon.json 文件，把以下内容复制/添加进去

JSON
{
  "insecure-registries": [
    "192.168.100.10"
  ]
}

​ 然后重新加载 daemon 并重新启动 Docker

BASH
$ sudo systemctl daemon-reload
$ sudo systemctl restart docker

​ 然后耐心等待 Jenkins 和 GitLab 的容器恢复

5.3 关闭 containerd 的 HTTPS 请求

​ 编辑 两台主机 的 /etc/containerd/config.toml 文件，修改或添加以下内容

TOML
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.100.10"]
  endpoint = ["http://192.168.100.10"]

​ 然后重启 containerd 服务

BASH
$ sudo systemctl restart containerd

6. 配置 CI/CD

6.1 新建任务

​ 登录 Jenkins Dashboard，点击 “新建Item”(新建任务)，选择构建一个流水线，然后 “确定”

2.6.1.1

​ 然后在 “构建触发器” 一栏，勾选 “Build when a change is pushed to GitLab. GitLab webhook URL: http://192.168.100.10:8080/project/ChinaskillProject↗”

​ 记录下 GitLab Webhook URL的地址(http://192.168.100.10:8080/project/ChinaskillProject)，后期配置 Webhook 需要使用

​ 然后配置下方 “流水线”，然后点击下面的 “流水线语法”：“示例步骤” 选择 “git：Git”，将 ChinaskillProject 项目的 GitLab 地址填入“仓库 URL”

​ 然后点一下 “凭据” 旁边的 “添加” 按钮，选择 “Jenkins”，类型选择 “Username with password”，用户名和密码为 GitLab 仓库的用户名和密码

2.6.1.2

​ 添加凭据之后选择凭据，最后是这样的

2.6.1.3

​ 然后点击 “生成流水线脚本”

2.6.1.4

​ 然后记录生成的值

TEXT
git credentialsId: '757b55dd-2337-4703-808d-bbf147610cd4', url: 'http://192.168.100.10:81/root/chinaskillproject.git'

​ 将其写入下面的流水线脚本里面

​ 这里是完整的流水线脚本

GROOVY
node {
    stage('git clone') {
        // Check CODE
        git credentialsId: '757b55dd-2337-4703-808d-bbf147610cd4', url: 'http://192.168.100.10:81/root/chinaskillproject.git'
    }
    stage('maven build') {
        sh '''/usr/local/maven/bin/mvn package -DskipTests -f /var/jenkins_home/workspace/ChinaskillProject'''
    }
    stage('image build') {
        sh '''
            echo $BUILD_ID
            docker build -t 192.168.100.10/chinaskillproject/gateway:$BUILD_ID -f /var/jenkins_home/workspace/ChinaskillProject/gateway/Dockerfile  /var/jenkins_home/workspace/ChinaskillProject/gateway
            docker build -t 192.168.100.10/chinaskillproject/config:$BUILD_ID -f /var/jenkins_home/workspace/ChinaskillProject/config/Dockerfile  /var/jenkins_home/workspace/ChinaskillProject/config
        '''
    }
    stage('test') {
        sh '''
            docker run -itd --name gateway 192.168.100.10/chinaskillproject/gateway:$BUILD_ID
            docker ps -a | grep chinaskillproject | grep Up
            if [ $? -eq 0 ]; then
                echo "Success!"
                docker rm -f gateway
            else
                docker rm -f gateway
                exit 1
            fi
        '''
    }
    stage('upload registry') {
        sh '''
            docker login 192.168.100.10 -u=admin -p=Harbor12345
            docker push 192.168.100.10/chinaskillproject/gateway:$BUILD_ID
            docker push 192.168.100.10/chinaskillproject/config:$BUILD_ID
        '''
    }
    stage('deploy Rancher') {
        // 执行部署脚本
        sh 'sed -i "s/sqshq\\/piggymetrics-gateway/192.168.100.10\\/chinaskillproject\\/gateway:$BUILD_ID/g" /var/jenkins_home/workspace/ChinaskillProject/yaml/deployment/gateway-deployment.yaml'
        sh 'sed -i "s/sqshq\\/piggymetrics-config/192.168.100.10\\/chinaskillproject\\/config:$BUILD_ID/g" /var/jenkins_home/workspace/ChinaskillProject/yaml/deployment/config-deployment.yaml'
        sh 'kubectl create ns ChinaskillProject'
        sh 'kubectl apply -f /var/jenkins_home/workspace/ChinaskillProject/yaml/deployment/gateway-deployment.yaml --kubeconfig=/root/.kube/config'
        sh 'kubectl apply -f /var/jenkins_home/workspace/ChinaskillProject/yaml/deployment/config-deployment.yaml --kubeconfig=/root/.kube/config'
        sh 'kubectl apply -f /var/jenkins_home/workspace/ChinaskillProject/yaml/svc/gateway-svc.yaml --kubeconfig=/root/.kube/config'
        sh 'kubectl apply -f /var/jenkins_home/workspace/ChinaskillProject/yaml/svc/config-svc.yaml --kubeconfig=/root/.kube/config'
    }
}

​ 在 “脚本” 一栏里写入脚本之后即可点击 “Save” 或者是 “应用”

2.6.1.5

6.2 开启 Jenkins 匿名访问

​ 登录 Jenkins Dashboard ，点击 “Manage Jenkins”(管理 Jenkins/系统管理) → “Security”(安全/全局安全配置)，然后配置授权策略允许匿名用户访问

2.6.2.1

6.3 配置 Webhook

​ 登录 GitLab，进入 “ChinaskillProject” 项目

​ 点击 “Settings” → “Webhooks”，将前面记录的 GitLab Webhook URL 地址填入 URL 处，并禁用 SSL 认证

2.6.3.1

​ 点击 “Add webhook” 添加 Webhook

2.6.3.2

​ 然后点击 “Test”(测试) → “Push events”(推送活动) 进行测试

2.6.3.3

​ 结果返回 HTTP 200 则表明 Webhook 配置成功

6.4 创建仓库项目

​ 登录 Harbor，新建项目 chinaskillproject，访问级别设置为 “公开”

2.6.4.1

​ 进入项目查看镜像列表，可以看到此时为空，无任何镜像

2.6.4.2

三、正式构建

触发 CI/CD

注意

​ 在此之前，需要把 XianDian-PaaS-v2.4.iso 里面的 ./ChinaskillProject 项目下所有目录里面的 Dockerfile 的 java:8-jre 全都改成 openjdk:8-jre，然后重新上传 GitLab 仓库，否则会导致构建失败

  • 解决方法之一：

    • 在修改完内容之后，新文件夹是 /root/ChinaskillProject

      BASH
      $ cd /root/ChinaskillProject
$ git remote remove origin
$ git remote add origin http://192.168.100.10:81/root/chinaskillproject.git
$ git add .
$ git commit -m "Initial commit"
$ git push -u origin master -f

    • 然后刷新 GitLab，看 commit 时间是最近的说明成功

    • 如果提示错误，就到 GitLab 的 ChinaskillProject 项目里，进入 “Settings”(设置) → ""

1. 触发构建

​ 回到 GitLab 上的 “ChinaskillProject” 项目，点击 “Settings” → “Webhooks” 往下翻找到 “Test”(测试) 然后点击 “Push events”(推送活动) 即可正常构建

​ 或者你也可以上传代码，来自动触发构建

BASH
$ docker cp /opt/repository/ jenkins:/root/.m2/
$ cd /opt/ChinaskillProject/
$ git add .
$ git commit -m "Initial commit"
$ git push -u origin master
Username for 'http://192.168.100.10:81': root
Password for 'http://root@192.168.100.10:81': 
Counting objects: 3068, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (1358/1358), done.
Writing objects: 100% (3068/3068), 1.39 MiB | 0 bytes/s, done.
Total 3068 (delta 1236), reused 3013 (delta 1207)
remote: Resolving deltas: 100% (1236/1236), done.
To http://192.168.100.10:81/root/chinaskillproject.git
 + 1b2901a...f44d317 master -> master (forced update)
2. Jenkins 查看

​ 登录 Jenkins Dashboard，可以看到 ChinaskillProject 项目已经开始构建

​ 点击项目名称查看流水线阶段视图，点击左边最新的#（我这边是#2）里面的 “Console Output”(控制台输出) 可以看到详细的构建进程

3.2.13.2.2

3. Harbor 查看

​ 进入 Harbor 仓库 chinaskillproject 项目查看镜像列表，可以看到已经自动上传了一个 gateway 和 config 镜像

3.3.1

4. Kubernertes 查看

​ Pod 的启动比较慢，需要等待 3~5 分钟，具体的时间需要看你分配的配置

​ 我们可以在命令行查看 Pod

BASH
$ kubectl -n ChinaskillProject get pods
NAME                       READY   STATUS    RESTARTS       AGE
config-54cf699d8d-6pvx7    1/1     Running   0              2m29s
gateway-855db6cd7d-vhsqx   1/1     Running   1 (2m7s ago)   2m30s

​ 查看 Service

BASH
$ kubectl -n ChinaskillProject get service
NAME      TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
config    NodePort   10.96.109.132    <none>        8888:30015/TCP   24m
gateway   NodePort   10.106.190.223   <none>        4000:30010/TCP   24m

​ 然后访问 http://192.168.100.10:30010 访问服务

3.4.1

Footnotes

  1. Docker + Kubernetes 可以根据这个脚本一键安装：AliYUM+Docker+K8s_v1.5.0_CN↗ ↩

  2. 如果不可以，可以试试这个脚本（*需要海外网络环境）：AliYUM+Docker+K8s_v1.5.0↗ ↩

  3. 如果不会搭建的可以查看【二、5】 ↩

  4. XianDian-PaaS-v2.4.iso 下载连接：https://hezhi.anitsuri.top/filebrowser/api/public/dl/TslUutnB/Share/XianDian-PaaS-v2.4.iso↗ ↩ ↩2

  5. 如果链接失效，请联系 E-mail 或 B站私信↗ ↩

Thanks for reading!

基于 Kubernetes 的 CI/CD

周五 12月 27 2024
4334 字 · 27 分钟
Documentation 云运维工坊 Kubernetes Harbor CI/CD DevOps Jenkins git

© AnitsuriW | CC BY-NC-SA 4.0

分享这篇博客

X (Twitter) Telegram Reddit Facebook Email

目录