容器编排技术

一、前言

了解Jenkins的离线安装步骤
掌握Gitlab的使用和管理
了解CICD的配置步骤和方法

二、基础环境部署

机器	IP	作用
master	192.168.100.10	K8s 的 master 节点、Harbor 节点、CI/CD 节点
node	192.168.100.20	K8s 的 node 节点

在这两台机器上提前部署好 CentOS 7 + Docker + Kubernetes¹² + Harbor³ 环境，准备好 XianDian-PaaS-v2.4.iso 包⁴⁵

1. 安装 Jenkins 环境

1.1 查看 K8s 集群状态和节点信息

 1
 2
 3
 4
 5
 6
 7
 8
 9
10


$ kubectl get cs
Warning: v1 ComponentStatus is deprecated in v1.19+
NAME                 STATUS    MESSAGE   ERROR
scheduler            Healthy   ok        
controller-manager   Healthy   ok        
etcd-0               Healthy
$ kubectl get nodes
NAME     STATUS   ROLES           AGE   VERSION
master   Ready    control-plane   31d   v1.27.16
node     Ready    <none>          31d   v1.27.16

1.2 安装 Jenkins

1
2
3
4
5
6
7
8


$  docker run -d --name jenkins -p 8080:8080 -u root \
-v /home/jenkins_home:/var/jenkins_home \
-v /var/run/docker.sock:/var/run/docker.sock \
-v $(which docker):/usr/bin/docker \
-v /usr/bin/kubectl:/usr/local/bin/kubectl \
-v /root/.kube:/root/.kube \
jenkins/jenkins:latest-jdk21
22f894f46083c2b8836ae4e4c67e0bf4e8cd2e09adcc29417d6b8b473dea1407

然后在电脑浏览器输入 http://192.168.100.10:8080 来访问 Jenkins

2.1.2.1

然后进入容器，输入密码

1
2


$ docker exec jenkins cat /var/jenkins_home/secrets/initialAdminPassword
2169d1cbaa1546f396c0571477d2ce6a

下面的 2169d1cbaa1546f396c0571477d2ce6a 就是我们的密码，输入密码并点击 “继续”

2.1.2.2

然后点击 “安装推荐的插件”

2.1.2.3

插件安装完成后自动转入管理员用户创建界面

2.1.2.4

输入用户信息，点击“保存并完成”

配置完URL后使用新建的用户登录 Jenkins

2.1.2.5

2. 部署 GitLab

2.1 启动 GitLab

GitLab 是利用 Ruby on Rails 一个开源的版本管理系统，实现一个自托管的 Git 项目仓库，可通过 Web 界面进行访问公开的或者私人项目。

与 Github 类似，GitLab 能够浏览源代码，管理缺陷和注释，可以管理团队对仓库的访问，它非常易于浏览提交过的版本并提供一个文件历史库，团队成员可以利用内置的简单聊天程序(Wall) 进行交流。Gitlab 还提供一个代码片段收集功能可以轻松实现代码复用，便于日后有需要的时候进行查找

本项目Gitlab与Harbor共用一台服务器

1
2
3
4
5
6


$ docker run -d -h gitlab -p 1022:22 -p 81:80 -p 443:443 \
--volume /srv/gitlab/config:/etc/gitlab \
--volume /srv/gitlab/gitlab/logs:/var/log/gitlab \
--volume /srv/gitlab/gitlab/data:/var/opt/gitlab \
--restart always --name mygitlab gitlab/gitlab-ce:latest
e87a1e5298f2c0430ad5f2269472e9f06cc2eddf71477fb28a6e8089cf46f8d4

GitLab 启动较慢，可以通过 docker logs 查看启动状态

1

$ docker logs mygitlab

启动完成后，在浏览器上访问 http://192.168.100.10:81 来进入刚刚搭建的本地 GitLab

然后设置 root 用户信息

2.2.1.1

然后使用 root 用户登录

2.2.1.2 2.2.1.3

2.2 创建项目

点击 “Create a project”(创建项目)，创建项目 “ChinaskillProject”，Visibility Level(可见等级) 设置为 “Public”

2.2.2.1

点击 “Create Project”(创建项目)，然后自动进入项目

2.2.2.2

根据提示，配置 SSH 公钥到 GitLab 上

1
2


$ ssh-keygen -t rsa
$ cat ~/.ssh/id_rsa.pub

然后复制出来的内容，到 GitLab 上

GitLab 上点击 “Add SSH Key”(添加 SSH 密钥)，在 Key 栏里面粘贴你刚才复制的内容，Title 默认即可，Expires at(有效期截至到) 随便选，然后点击 “Add Key”(添加密钥)

2.2.2.3

然后将之前准备好的 XianDian-PaaS-v2.4.iso⁴ 上传并挂载到虚拟机， git push 源代码到 GitLab 的 ChinaskillProject 项目

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48


$ sudo yum install -y git
$ mount /root/XianDian-PaaS-v2.4.iso /mnt/
$ cp -rf /mnt/ChinaskillProject/ /opt
$ ls -al /opt/ChinaskillProject/
总用量 36
dr-xr-xr-x  13 root root 4096 12月 25 11:18 .
drwxr-xr-x.  5 root root   60 12月 25 11:18 ..
dr-xr-xr-x   3 root root   50 12月 25 11:18 account-service
dr-xr-xr-x   3 root root   50 12月 25 11:18 auth-service
dr-xr-xr-x   3 root root   50 12月 25 11:18 config
-r-xr-xr-x   1 root root  823 12月 25 11:18 docker-compose.dev.yml
-r-xr-xr-x   1 root root 4020 12月 25 11:18 docker-compose.yml
dr-xr-xr-x   3 root root   50 12月 25 11:18 gateway
dr-xr-xr-x   8 root root  185 12月 25 11:18 .git
-r-xr-xr-x   1 root root   68 12月 25 11:18 .gitignore
-r-xr-xr-x   1 root root 1107 12月 25 11:18 LICENCE
dr-xr-xr-x   3 root root   51 12月 25 11:18 mongodb
dr-xr-xr-x   3 root root   50 12月 25 11:18 monitoring
dr-xr-xr-x   3 root root   50 12月 25 11:18 notification-service
-r-xr-xr-x   1 root root  723 12月 25 11:18 pom.xml
-r-xr-xr-x   1 root root 1075 12月 25 11:18 README.md
dr-xr-xr-x   3 root root   50 12月 25 11:18 registry
dr-xr-xr-x   3 root root   50 12月 25 11:18 statistics-service
-r-xr-xr-x   1 root root 4428 12月 25 11:18 .travis.yml
dr-xr-xr-x   4 root root   35 12月 25 11:18 yaml

$ cd /opt/ChinaskillProject/
# 移除“指定远程目录”
$ git remote remove origin
# 指定“我们自己的”远程目录
$ git remote add origin http://192.168.100.10:81/root/chinaskillproject.git

$ git add .
$ git commit -m "Initial commit"
$ git push -u origin master
# 这里输入 root
Username for 'http://192.168.100.10:81': root
# 这里设置你刚才配置 GitLab 的时候配置的密码
Password for 'http://root@192.168.100.10:81': 
Counting objects: 3192, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (1428/1428), done.
Writing objects: 100% (3192/3192), 1.40 MiB | 0 bytes/s, done.
Total 3192 (delta 1233), reused 3010 (delta 1207)
remote: Resolving deltas: 100% (1233/1233), done.
To http://192.168.100.10:81/root/chinaskillproject.git
 * [new branch]      master -> master
分支 master 设置为跟踪来自 origin 的远程分支 master。

刷新网页，ChinaskillProject 项目中已经有我们刚刚 git push 上去的文件了

2.2.2.4

3. 配置 Jenkins 连接 GitLab

3.1 设置 Outbound requests(出站请求)

然后点击 “Settings” → “Network” → “Outbound requests”(出战请求) 旁边的 Expand(展开)

然后勾选 “Allow requests to the local network from web hooks and services”

在下面的输入栏里面输入 192.168.100.0/24

然后点击 “Save change”(保存设置)

2.3.1.1

3.2 创建 Github API Token

点击 GitLab 的用户头像，然后点击 “Settings” → “Access Token” 添加一个 Token

在 Scopes(范围) 上选上 “api” 和 “read_user”，其他的和上面一样，随便填一个就行，然后点击 “Create personal access token”(创建个人访问令牌)

2.3.2.1

然后复制 “Your New Personal Access Token”(你的新个人访问令牌) 栏里面的内容，后面配置 Jenkins 的时候需要用到

1

bWxAexndVn6N-XsmFHWz

2.3.2.2

3.3 设置 Jenkins

“GitLab”
“Generic Webhook Trigger”
“GitLab API”
“GitLab Authentication”
“GitLab Branch Source”
“Gitlab Merge Request Builder”
“GitLab Logo”

如果没有的话就到 “Available plugins”(可用插件) 去安装一下

然后回到 Jenkins Dashboard，点击 “Manage Jenkins”(管理 Jenkins/系统管理) → “System”(系统配置)

然后配置 GitLab 信息，取消勾选 “Enable authentication for ‘/project’ end-point” 然后在 “Credentials” 下面点击 “添加”

然后在类型里面选择 “GitLab API token”，“API token” 填入刚刚复制的 token

然后点击 “Test Connection”(测试连接)

2.3.3.1 2.3.3.2

4. 配置 Jenkins 连接 maven

4.1 安装 maven

由于我们这里的 Jenkins 是采用 Docker in Docker 的方式启动的，所以需要在 Jenkins 容器内安装 maven

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


# 下载 maven-3.6.3
$ curl -o /home/jenkins_home/apache-maven-3.6.3-bin.tar.gz https://archive.apache.org/dist/maven/maven-3/3.6.3/binaries/apache-maven-3.6.3-bin.tar.gz

# 进入 Docker 容器
$ docker exec -it jenkins /bin/bash
	$ tar -zxvf /var/jenkins_home/apache-maven-3.6.3-bin.tar.gz -C .
	$ mv apache-maven-3.6.3/ /usr/local/maven
	$ apt update -y && apt upgrade -y
	$ apt install -y vim
	$ vim ~/.bashrc
	# 文末加入
	export M2_HOME=/usr/local/maven
	export PATH=$PATH:$M2_HOME/bin
	
	$ source ~/.bashrc
	$ exit
# 退出容器重新进入
$ docker exec -it jenkins /bin/bash
	$ mvn -v
	Apache Maven 3.6.3 (cecedd343002696d0abb50b32b541b8a6ba2883f)
	Maven home: /usr/local/maven
	Java version: 21.0.5, vendor: Eclipse Adoptium, runtime: /opt/java/openjdk
	Default locale: en, platform encoding: UTF-8
	OS name: "linux", version: "3.10.0-1160.71.1.el7.x86_64", arch: "amd64", family: "unix"

4.2 连接 maven

往下翻找到 Maven，点击“新增Maven”，此处不要开启自动安装，所以不要勾选“自动安装”

然后把填入 Maven 路径填入 MAVEN_HOME ，应用保存即可

2.4.2.1

5. 部署 Harbor 仓库

5.1 安装 Harbor 仓库

Harbor 依赖 Docker 和 Docker Compose。因此，首先需要在系统中安装 Docker Compose

1
2
3
4
5


$ DOCKER_CONFIG=/usr/local/lib/docker/cli-plugins
$ sudo mkdir -p $DOCKER_CONFIG/cli-plugins
$ sudo curl -SL https://github.com/docker/compose/releases/download/v2.29.1/docker-compose-linux-x86_64 -o $DOCKER_CONFIG/cli-plugins/docker-compose
$ sudo chmod +x $DOCKER_CONFIG/cli-plugins
$ docker compose version

然后下载 Offline 版的 Harbor

1
2
3
4
5
6
7


$ sudo curl -O https://github.com/goharbor/harbor/releases/download/v2.12.1/harbor-offline-installer-v2.12.1.tgz

# 国内用户可以使用以下方式加快下载
$ sudo curl -O https://moka.anitsuri.top/images/Registry+Harbor/harbor-offline-installer-v2.11.1.tgz

$ sudo tar -zxf harbor-offline-installer-v2.12.1.tgz
$ cd harbor

然后修改配置文件，把 harbor.yml.tmpl 复制一个变成 harbor.yml

1
2


$ sudo cp harbor.yml.tmpl harbor.yml
$ sudo vi harbor.yml

修改/填写以下内容

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47


# Configuration file of Harbor

# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: 192.168.100.10

# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 80

# https related config
# https:
  # https port for harbor, default is 443
  # port: 443
  # The path of cert and key files for nginx
  # certificate: /your/certificate/path
  # private_key: /your/private/key/path
  # enable strong ssl ciphers (default: false)
  # strong_ssl_ciphers: false

# # Harbor will set ipv4 enabled only by default if this block is not configured
# # Otherwise, please uncomment this block to configure your own ip_family stacks
# ip_family:
#   # ipv6Enabled set to true if ipv6 is enabled in docker network, currently it affected the nginx related component
#   ipv6:
#     enabled: false
#   # ipv4Enabled set to true by default, currently it affected the nginx related component
#   ipv4:
#     enabled: true

# # Uncomment following will enable tls communication between all harbor components
# internal_tls:
#   # set enabled to true means internal tls is enabled
#   enabled: true
#   # put your cert and key files on dir
#   dir: /etc/harbor/tls/internal


# Uncomment external_url if you want to enable external proxy
# And when it enabled the hostname will no longer used
# external_url: http://192.168.92.128:8433

# The initial password of Harbor admin
# It only works in first time to install harbor
# Remember Change the admin password from UI after launching Harbor.
harbor_admin_password: Harbor12345

在运行安装脚本前，需要先运行 prepare 脚本，来确保一切依赖和配置都已经就绪

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172


$ ./prepare
prepare base dir is set to /root/harbor
Unable to find image 'goharbor/prepare:v2.12.1' locally
v2.12.1: Pulling from goharbor/prepare
0e5c20819ebf: Pull complete 
1fb69a2a4ee3: Pull complete 
c2aef12efe8a: Pull complete 
79fdf0882382: Pull complete 
eee2f1d222f1: Pull complete 
125a453a0ce4: Pull complete 
40b43804af0b: Pull complete 
5979e246bb31: Pull complete 
8d2c370a2c28: Pull complete 
53eaa20ee0df: Pull complete 
Digest: sha256:d0fcec61328185ddb5bfded6bf8b312428aa63b9bffe2a8d846452253234a898
Status: Downloaded newer image for goharbor/prepare:v2.12.1
WARNING:root:WARNING: HTTP protocol is insecure. Harbor will deprecate http protocol in the future. Please make sure to upgrade to https
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
Generated and saved secret to file: /data/secret/keys/secretkey
Successfully called func: create_root_cert
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir

$ ./install.sh

[Step 0]: checking if docker is installed ...

Note: docker version: 26.1.4

[Step 1]: checking docker-compose is installed ...

Note: Docker Compose version v2.27.1

[Step 2]: loading Harbor images ...
581136816168: Loading layer [==================================================>]  17.21MB/17.21MB
cbf4f3cb0c26: Loading layer [==================================================>]  3.584kB/3.584kB
19ba001b8b03: Loading layer [==================================================>]   2.56kB/2.56kB
b41af00dcd76: Loading layer [==================================================>]  70.97MB/70.97MB
5b5f57643342: Loading layer [==================================================>]  5.632kB/5.632kB
5175984e7e7a: Loading layer [==================================================>]  126.5kB/126.5kB
c8faf279c61c: Loading layer [==================================================>]  201.7kB/201.7kB
3a0151c1eba9: Loading layer [==================================================>]  72.09MB/72.09MB
dbf86e000945: Loading layer [==================================================>]   2.56kB/2.56kB
Loaded image: goharbor/harbor-core:v2.12.1
7d2be923c4b3: Loading layer [==================================================>]  146.8MB/146.8MB
1b7e2d8bb9c9: Loading layer [==================================================>]  3.584kB/3.584kB
c433c2b34c5a: Loading layer [==================================================>]  3.072kB/3.072kB
902158d6f915: Loading layer [==================================================>]   2.56kB/2.56kB
e795d71404bd: Loading layer [==================================================>]  3.072kB/3.072kB
cec684dd57a5: Loading layer [==================================================>]  3.584kB/3.584kB
47d0e4a2c5c7: Loading layer [==================================================>]  20.48kB/20.48kB
Loaded image: goharbor/harbor-log:v2.12.1
594f33e86793: Loading layer [==================================================>]  22.04MB/22.04MB
e0a4bafe73dc: Loading layer [==================================================>]  191.7MB/191.7MB
26b746af8db7: Loading layer [==================================================>]  26.16MB/26.16MB
31dcafd9cb03: Loading layer [==================================================>]  18.54MB/18.54MB
abf08c52686b: Loading layer [==================================================>]   5.12kB/5.12kB
0007d565ec69: Loading layer [==================================================>]  6.144kB/6.144kB
6e863404971d: Loading layer [==================================================>]  3.072kB/3.072kB
94c8eba384cc: Loading layer [==================================================>]  2.048kB/2.048kB
1a5955f6795e: Loading layer [==================================================>]   2.56kB/2.56kB
8ce6cc317185: Loading layer [==================================================>]   7.68kB/7.68kB
Loaded image: goharbor/harbor-db:v2.12.1
71775c4c621c: Loading layer [==================================================>]  17.21MB/17.21MB
4003ab14e211: Loading layer [==================================================>]  3.584kB/3.584kB
beabc71b1223: Loading layer [==================================================>]   2.56kB/2.56kB
579b2f017459: Loading layer [==================================================>]   59.9MB/59.9MB
895f989740a0: Loading layer [==================================================>]  60.69MB/60.69MB
Loaded image: goharbor/harbor-jobservice:v2.12.1
9590068fe044: Loading layer [==================================================>]  14.26MB/14.26MB
0cb250c20318: Loading layer [==================================================>]  4.096kB/4.096kB
9dac56331be1: Loading layer [==================================================>]  18.14MB/18.14MB
5dc6f164dc78: Loading layer [==================================================>]  3.072kB/3.072kB
141d394a5d47: Loading layer [==================================================>]  37.13MB/37.13MB
2587acb702ad: Loading layer [==================================================>]  56.05MB/56.05MB
Loaded image: goharbor/harbor-registryctl:v2.12.1
dc1a7f40e575: Loading layer [==================================================>]  17.21MB/17.21MB
3f56a5b9b5d4: Loading layer [==================================================>]  37.15MB/37.15MB
d67a91463720: Loading layer [==================================================>]  4.608kB/4.608kB
cedc4b1b129a: Loading layer [==================================================>]  37.94MB/37.94MB
Loaded image: goharbor/harbor-exporter:v2.12.1
37959c9a5eac: Loading layer [==================================================>]  137.2MB/137.2MB
Loaded image: goharbor/nginx-photon:v2.12.1
d07b5c3ce0a9: Loading layer [==================================================>]  137.2MB/137.2MB
f60722775a24: Loading layer [==================================================>]  6.731MB/6.731MB
4c9a05314a30: Loading layer [==================================================>]  252.9kB/252.9kB
61258323c496: Loading layer [==================================================>]  1.497MB/1.497MB
Loaded image: goharbor/harbor-portal:v2.12.1
9a1a8734fa3c: Loading layer [==================================================>]  14.75MB/14.75MB
856558c960ff: Loading layer [==================================================>]  4.096kB/4.096kB
9bd63d8a60d9: Loading layer [==================================================>]  3.072kB/3.072kB
a89a7d8b1691: Loading layer [==================================================>]  133.9MB/133.9MB
008e693070f3: Loading layer [==================================================>]  15.55MB/15.55MB
cef35bccdc97: Loading layer [==================================================>]  150.3MB/150.3MB
Loaded image: goharbor/trivy-adapter-photon:v2.12.1
d17edfb68476: Loading layer [==================================================>]  22.06MB/22.06MB
a0f8af0fdb08: Loading layer [==================================================>]  126.6MB/126.6MB
27e661303e23: Loading layer [==================================================>]  3.072kB/3.072kB
3f0390c3788e: Loading layer [==================================================>]   59.9kB/59.9kB
f56e18460c59: Loading layer [==================================================>]  61.95kB/61.95kB
Loaded image: goharbor/redis-photon:v2.12.1
8b7f3238d8aa: Loading layer [==================================================>]  14.26MB/14.26MB
295711278366: Loading layer [==================================================>]  4.096kB/4.096kB
ddadeb03375d: Loading layer [==================================================>]  3.072kB/3.072kB
99bbf20549c8: Loading layer [==================================================>]  18.14MB/18.14MB
9237fc270c32: Loading layer [==================================================>]  18.93MB/18.93MB
Loaded image: goharbor/registry-photon:v2.12.1
Loaded image: goharbor/prepare:v2.12.1


[Step 3]: preparing environment ...

[Step 4]: preparing harbor configs ...
prepare base dir is set to /root/harbor
WARNING:root:WARNING: HTTP protocol is insecure. Harbor will deprecate http protocol in the future. Please make sure to upgrade to https
Clearing the configuration file: /config/portal/nginx.conf
Clearing the configuration file: /config/log/logrotate.conf
Clearing the configuration file: /config/log/rsyslog_docker.conf
Clearing the configuration file: /config/nginx/nginx.conf
Clearing the configuration file: /config/core/env
Clearing the configuration file: /config/core/app.conf
Clearing the configuration file: /config/registry/passwd
Clearing the configuration file: /config/registry/config.yml
Clearing the configuration file: /config/registryctl/env
Clearing the configuration file: /config/registryctl/config.yml
Clearing the configuration file: /config/db/env
Clearing the configuration file: /config/jobservice/env
Clearing the configuration file: /config/jobservice/config.yml
Generated configuration file: /config/portal/nginx.conf
Generated configuration file: /config/log/logrotate.conf
Generated configuration file: /config/log/rsyslog_docker.conf
Generated configuration file: /config/nginx/nginx.conf
Generated configuration file: /config/core/env
Generated configuration file: /config/core/app.conf
Generated configuration file: /config/registry/config.yml
Generated configuration file: /config/registryctl/env
Generated configuration file: /config/registryctl/config.yml
Generated configuration file: /config/db/env
Generated configuration file: /config/jobservice/env
Generated configuration file: /config/jobservice/config.yml
loaded secret from file: /data/secret/keys/secretkey
Generated configuration file: /compose_location/docker-compose.yml
Clean up the input dir


Note: stopping existing Harbor instance ...


[Step 5]: starting Harbor ...
[+] Running 10/10
 ✔ Network harbor_harbor        Created                                                                                    0.3s 
 ✔ Container harbor-log         Started                                                                                    2.0s 
 ✔ Container registryctl        Started                                                                                    2.9s 
 ✔ Container redis              Started                                                                                    2.9s 
 ✔ Container harbor-db          Started                                                                                    3.0s 
 ✔ Container harbor-portal      Started                                                                                    3.0s 
 ✔ Container registry           Started                                                                                    3.0s 
 ✔ Container harbor-core        Started                                                                                    3.2s 
 ✔ Container nginx              Started                                                                                    3.8s 
 ✔ Container harbor-jobservice  Started                                                                                    3.7s 
✔ ----Harbor has been installed and started successfully.----

5.2 配置 Harbor 到 Docker Daemon 文件

在两台机器上，编辑 /etc/docker/daemon.json 文件，把以下内容复制/添加进去

1
2
3
4
5


{
  "insecure-registries": [
    "192.168.100.10"
  ]
}

然后重新加载 daemon 并重新启动 Docker

1
2


$ sudo systemctl daemon-reload
$ sudo systemctl restart docker

然后耐心等待 Jenkins 和 GitLab 的容器恢复

5.3 关闭 containerd 的 HTTPS 请求

编辑 两台主机 的 /etc/containerd/config.toml 文件，修改或添加以下内容

1
2


[plugins."io.containerd.grpc.v1.cri".registry.mirrors."192.168.100.10"]
  endpoint = ["http://192.168.100.10"]

然后重启 containerd 服务

1

$ sudo systemctl restart containerd

6. 配置 CI/CD

6.1 新建任务

2.6.1.1

然后在 “构建触发器” 一栏，勾选 “Build when a change is pushed to GitLab. GitLab webhook URL: http://192.168.100.10:8080/project/ChinaskillProject”

记录下 GitLab Webhook URL的地址(http://192.168.100.10:8080/project/ChinaskillProject)，后期配置 Webhook 需要使用

然后配置下方 “流水线”，然后点击下面的 “流水线语法”：“示例步骤” 选择 “git：Git”，将 ChinaskillProject 项目的 GitLab 地址填入“仓库 URL”

然后点一下 “凭据” 旁边的 “添加” 按钮，选择 “Jenkins”，类型选择 “Username with password”，用户名和密码为 GitLab 仓库的用户名和密码

2.6.1.2

添加凭据之后选择凭据，最后是这样的

2.6.1.3

然后点击 “生成流水线脚本”

2.6.1.4

然后记录生成的值

1

git credentialsId: '757b55dd-2337-4703-808d-bbf147610cd4', url: 'http://192.168.100.10:81/root/chinaskillproject.git'

将其写入下面的流水线脚本里面

这里是完整的流水线脚本

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46


node {
    stage('git clone') {
        // Check CODE
        git credentialsId: '757b55dd-2337-4703-808d-bbf147610cd4', url: 'http://192.168.100.10:81/root/chinaskillproject.git'
    }
    stage('maven build') {
        sh '''/usr/local/maven/bin/mvn package -DskipTests -f /var/jenkins_home/workspace/ChinaskillProject'''
    }
    stage('image build') {
        sh '''
            echo $BUILD_ID
            docker build -t 192.168.100.10/chinaskillproject/gateway:$BUILD_ID -f /var/jenkins_home/workspace/ChinaskillProject/gateway/Dockerfile  /var/jenkins_home/workspace/ChinaskillProject/gateway
            docker build -t 192.168.100.10/chinaskillproject/config:$BUILD_ID -f /var/jenkins_home/workspace/ChinaskillProject/config/Dockerfile  /var/jenkins_home/workspace/ChinaskillProject/config
        '''
    }
    stage('test') {
        sh '''
            docker run -itd --name gateway 192.168.100.10/chinaskillproject/gateway:$BUILD_ID
            docker ps -a | grep chinaskillproject | grep Up
            if [ $? -eq 0 ]; then
                echo "Success!"
                docker rm -f gateway
            else
                docker rm -f gateway
                exit 1
            fi
        '''
    }
    stage('upload registry') {
        sh '''
            docker login 192.168.100.10 -u=admin -p=Harbor12345
            docker push 192.168.100.10/chinaskillproject/gateway:$BUILD_ID
            docker push 192.168.100.10/chinaskillproject/config:$BUILD_ID
        '''
    }
    stage('deploy Rancher') {
        // 执行部署脚本
        sh 'sed -i "s/sqshq\\/piggymetrics-gateway/192.168.100.10\\/chinaskillproject\\/gateway:$BUILD_ID/g" /var/jenkins_home/workspace/ChinaskillProject/yaml/deployment/gateway-deployment.yaml'
        sh 'sed -i "s/sqshq\\/piggymetrics-config/192.168.100.10\\/chinaskillproject\\/config:$BUILD_ID/g" /var/jenkins_home/workspace/ChinaskillProject/yaml/deployment/config-deployment.yaml'
        sh 'kubectl create ns ChinaskillProject'
        sh 'kubectl apply -f /var/jenkins_home/workspace/ChinaskillProject/yaml/deployment/gateway-deployment.yaml --kubeconfig=/root/.kube/config'
        sh 'kubectl apply -f /var/jenkins_home/workspace/ChinaskillProject/yaml/deployment/config-deployment.yaml --kubeconfig=/root/.kube/config'
        sh 'kubectl apply -f /var/jenkins_home/workspace/ChinaskillProject/yaml/svc/gateway-svc.yaml --kubeconfig=/root/.kube/config'
        sh 'kubectl apply -f /var/jenkins_home/workspace/ChinaskillProject/yaml/svc/config-svc.yaml --kubeconfig=/root/.kube/config'
    }
}

在 “脚本” 一栏里写入脚本之后即可点击 “Save” 或者是 “应用”

2.6.1.5

6.2 开启 Jenkins 匿名访问

2.6.2.1

6.3 配置 Webhook

点击 “Settings” → “Webhooks”，将前面记录的 GitLab Webhook URL 地址填入 URL 处，并禁用 SSL 认证

2.6.3.1

点击 “Add webhook” 添加 Webhook

2.6.3.2

然后点击 “Test”(测试) → “Push events”(推送活动) 进行测试

2.6.3.3

结果返回 HTTP 200 则表明 Webhook 配置成功

6.4 创建仓库项目

2.6.4.1

进入项目查看镜像列表，可以看到此时为空，无任何镜像

2.6.4.2

三、正式构建

触发 CI/CD

注意

在此之前，需要把 XianDian-PaaS-v2.4.iso 里面的 ./ChinaskillProject 项目下所有目录里面的 Dockerfile 的 java:8-jre 全都改成 openjdk:8-jre，然后重新上传 GitLab 仓库，否则会导致构建失败

解决方法之一：

在修改完内容之后，新文件夹是 /root/ChinaskillProject

1
2
3
4
5
6


$ cd /root/ChinaskillProject
$ git remote remove origin
$ git remote add origin http://192.168.100.10:81/root/chinaskillproject.git
$ git add .
$ git commit -m "Initial commit"
$ git push -u origin master -f

然后刷新 GitLab，看 commit 时间是最近的说明成功
如果提示错误，就到 GitLab 的 ChinaskillProject 项目里，进入 “Settings”(设置) → ""

1. 触发构建

回到 GitLab 上的 “ChinaskillProject” 项目，点击 “Settings” → “Webhooks” 往下翻找到 “Test”(测试) 然后点击 “Push events”(推送活动) 即可正常构建

或者你也可以上传代码，来自动触发构建

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15


$ docker cp /opt/repository/ jenkins:/root/.m2/
$ cd /opt/ChinaskillProject/
$ git add .
$ git commit -m "Initial commit"
$ git push -u origin master
Username for 'http://192.168.100.10:81': root
Password for 'http://root@192.168.100.10:81': 
Counting objects: 3068, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (1358/1358), done.
Writing objects: 100% (3068/3068), 1.39 MiB | 0 bytes/s, done.
Total 3068 (delta 1236), reused 3013 (delta 1207)
remote: Resolving deltas: 100% (1236/1236), done.
To http://192.168.100.10:81/root/chinaskillproject.git
 + 1b2901a...f44d317 master -> master (forced update)

2. Jenkins 查看

点击项目名称查看流水线阶段视图，点击左边最新的#（我这边是#2）里面的 “Console Output”(控制台输出) 可以看到详细的构建进程

3.2.1 3.2.2

3. Harbor 查看

进入 Harbor 仓库 chinaskillproject 项目查看镜像列表，可以看到已经自动上传了一个 gateway 和 config 镜像

3.3.1

4. Kubernertes 查看

Pod 的启动比较慢，需要等待 3~5 分钟，具体的时间需要看你分配的配置

我们可以在命令行查看 Pod

1
2
3
4


$ kubectl -n ChinaskillProject get pods
NAME                       READY   STATUS    RESTARTS       AGE
config-54cf699d8d-6pvx7    1/1     Running   0              2m29s
gateway-855db6cd7d-vhsqx   1/1     Running   1 (2m7s ago)   2m30s

查看 Service

1
2
3
4


$ kubectl -n ChinaskillProject get service
NAME      TYPE       CLUSTER-IP       EXTERNAL-IP   PORT(S)          AGE
config    NodePort   10.96.109.132    <none>        8888:30015/TCP   24m
gateway   NodePort   10.106.190.223   <none>        4000:30010/TCP   24m

然后访问 http://192.168.100.10:30010 访问服务

3.4.1

Docker + Kubernetes 可以根据这个脚本一键安装：AliYUM+Docker+K8s_v1.5.0_CN ↩︎
如果不可以，可以试试这个脚本（*需要海外网络环境）：AliYUM+Docker+K8s_v1.5.0 ↩︎
如果不会搭建的可以查看【二、5】 ↩︎
XianDian-PaaS-v2.4.iso 下载连接：https://hezhi.anitsuri.top/filebrowser/api/public/dl/TslUutnB/Share/XianDian-PaaS-v2.4.iso ↩︎ ↩︎
如果链接失效，请联系 E-mail 或 B站私信 ↩︎

基于 Kubernetes 的 CI/CD

模拟实际生产环境部署离线 Kubernetes 和 CI/CD 环境，并测试