全離線環境下實現GitLabCI與Kubernetes對接

ARM 2040瀏覽

環境準備

本文將構建一個包含4臺工作機器的集群來模擬全離線環境下的GitLabCI環境及Kubernetes集群環境。相應的硬件配置如下:

虛擬機zz_z_gitlab:

8 Core, 16G, 200G Disk
gitlab IP: 192.168.122.160/24

虛擬機zz_z_gitlab_runner1, zz_z_gitlab_runner2, zz_z_gitlab_runner3:

8 Core, 16G, 200G Disk
runner1 IP: 192.168.122.161/24
runner2 IP: 192.168.122.162/24
runner3 IP: 192.168.122.163/24

說明:
gitlab比較費內存,因而節點分配的計算資源相對來說需要高一點。
以上的配置是在服務器上,因而沒有考慮到最小資源分配問題,在個人電腦上,因為內存和CPU的限制,可以使用2核4G的節點用來搭建Gitlab,2核4G用來搭建k8s單節點集群做實驗。

操作系統:采用Ubuntu16.04作為環境的推薦系統。在其上部署KubeSpray部署的Kubernetes集群。操作系統的安裝及Kubernetes的搭建并不在本文范疇內。

Kubernetes環境截圖:

http://www.sqtpkw.live/wp-content/uploads/images/201901/2019010216411517f29.jpeg

Registry準備(可選)

離線情況下,需要事先將Kubernetes用到的容器鏡像,以及本文gitlabci中用到的鏡像全部離線放入到私有倉庫,本文中采用的容器鏡像私有倉庫方案是opensuse提供的Portus,
以容器的方式運行于zz_z_gitlab節點上,當然你也可以使用gitlab自帶的registry服務來實現, 或者使用外置的harbor等方式。

Portus與GitLab存在端口重復的問題,需要在Portus的配置文件中,將默認監聽的80端口改為其他端口,如81端口。

離線安裝文件準備

需要實現下載gitlab的安裝文件(deb格式):

 gitlab-ce_11.0.3-ce.0_amd64.deb
 gitlab-runner_11.0.2_amd64.deb

安裝gitlab-runner時需要用到helm v2.7.0文件:

# wget https://kubernetes-helm.storage.googleapis.com/helm-v2.7.0-linux-amd64.tar.gz

離線容器準備

gitlab機器上,添加docker的insecure-registries方式:

# vim  /etc/docker/daemon.json
    {
        "insecure-registries" : ["portus.ooooooo.com:5000"]
    }
# systemctl restart docker
# docker login http://portus.ooooooo.com:5000
Username: kubespray
Password: 

alpine:3.6鏡像:

# docker tag alpine:3.6 portus.ooooooo.com:5000/kubesprayns/alpine:3.6
# docker push portus.ooooooo.com:5000/kubesprayns/alpine:3.6

http://www.sqtpkw.live/wp-content/uploads/images/201901/20190102164116fbbfb.jpeg

用于install helm的alpine鏡像準備:

$ sudo docker run -it alpine:3.6 /bin/sh
/ # apk add -U wget ca-certificates openssl >/dev/null
/ # which wget
/usr/bin/wget
/ # ls -l -h /usr/bin/wget
-rwxr-xr-x    1 root     root      443.2K May 11 13:02 /usr/bin/wget

保存容器,并上傳:

$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS               NAMES
79923690bcc2        alpine:3.6          "/bin/sh"           About a minute ago   Up About a minute                       jolly_banach
$ sudo docker commit 79923690bcc2 alpinewithwget:3.6
# docker tag alpinewithwget:3.6 portus.ooooooo.com:5000/kubesprayns/alpinewithwget:3.6
# docker push portus.ooooooo.com:5000/kubesprayns/alpinewithwget:3.6

helm在初始化的時候需要用到

$ sudo docker pull gcr.io/kubernetes-helm/tiller:v2.7.0
$ sudo docker tag gcr.io/kubernetes-helm/tiller:v2.7.0  portus.ooooooo.com:5000/kubesprayns/gcr.io/kubernetes-helm/tiller:v2.7.0 
$ sudo docker push portus.ooooooo.com:5000/kubesprayns/gcr.io/kubernetes-helm/tiller:v2.7.0 

gitlab-runner所需鏡像:

$ sudo docker pull busybox:latest
$ sudo docker pull ubuntu:16.04
$ sudo docker pull gitlab/gitlab-runner:alpine-v10.3.0
$ sudo docker pull gitlab/gitlab-runner-helper:x86_64-latest
$ sudo docker tag busybox:latest portus.ooooooo.com:5000/kubesprayns/busybox:latest
$ sudo docker tag ubuntu:16.04 portus.ooooooo.com:5000/kubesprayns/ubuntu:16.04
$ sudo docker tag gitlab/gitlab-runner:alpine-v10.3.0 portus.ooooooo.com:5000/kubesprayns/gitlab/gitlab-runner:alpine-v10.3.0
$ sudo docker tag gitlab/gitlab-runner-helper:x86_64-latest portus.ooooooo.com:5000/kubesprayns/gitlab/gitlab-runner-helper:x86_64-latest
$ sudo docker push portus.ooooooo.com:5000/kubesprayns/busybox:latest
$ sudo docker push portus.ooooooo.com:5000/kubesprayns/ubuntu:16.04
$ sudo docker push portus.ooooooo.com:5000/kubesprayns/gitlab/gitlab-runner:alpine-v10.3.0
$ sudo docker push portus.ooooooo.com:5000/kubesprayns/gitlab/gitlab-runner-helper:x86_64-latest

nginx-ingress-controller所需鏡像:

# docker pull quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.19.0
# docker pull k8s.gcr.io/defaultbackend:1.4
# docker tag quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.19.0 portus.ooooooo.com:5000/kubesprayns/quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.19.0
# docker push portus.ooooooo.com:5000/kubesprayns/quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.19.0
# docker tag k8s.gcr.io/defaultbackend:1.4 portus.ooooooo.com:5000/kubesprayns/k8s.gcr.io/defaultbackend:1.4
# docker push portus.ooooooo.com:5000/kubesprayns/k8s.gcr.io/defaultbackend:1.4

Prometheus所需容器:

$ sudo docker pull prom/alertmanager:v0.14.0
$ sudo docker pull jimmidyson/configmap-reload:v0.1
$ sudo docker pull quay.io/coreos/kube-state-metrics:v1.3.1
$ sudo docker pull prom/node-exporter:v0.15.2
$ sudo docker pull prom/prometheus:v2.2.1
$ sudo docker pull prom/prometheus:v2.1.0
$ sudo docker pull prom/pushgateway:v0.5.1
$ sudo docker pull prom/alertmanager:v0.14.0
$ sudo docker tag prom/alertmanager:v0.14.0 portus.oooooo:5000/kubesprayns/prom/alertmanager:v0.14.0
$ sudo docker tag jimmidyson/configmap-reload:v0.1 portus.oooooo:5000/kubesprayns/jimmidyson/configmap-reload:v0.1
$ sudo docker tag quay.io/coreos/kube-state-metrics:v1.3.1 portus.oooooo:5000/kubesprayns/quay.io/coreos/kube-state-metrics:v1.3.1
$ sudo docker tag prom/node-exporter:v0.15.2 portus.oooooo:5000/kubesprayns/prom/node-exporter:v0.15.2
$ sudo docker tag prom/prometheus:v2.2.1 portus.oooooo:5000/kubesprayns/prom/prometheus:v2.2.1
$ sudo docker tag prom/prometheus:v2.1.0 portus.oooooo:5000/kubesprayns/prom/prometheus:v2.1.0
$ sudo docker tag prom/pushgateway:v0.5.1 portus.oooooo:5000/kubesprayns/prom/pushgateway:v0.5.1
$ sudo docker tag prom/alertmanager:v0.14.0 portus.oooooo:5000/kubesprayns/prom/alertmanager:v0.14.0
$ sudo docker push  portus.oooooo:5000/kubesprayns/prom/alertmanager:v0.14.0
$ sudo docker push  portus.oooooo:5000/kubesprayns/jimmidyson/configmap-reload:v0.1
$ sudo docker push  portus.oooooo:5000/kubesprayns/quay.io/coreos/kube-state-metrics:v1.3.1
$ sudo docker push  portus.oooooo:5000/kubesprayns/prom/node-exporter:v0.15.2
$ sudo docker push  portus.oooooo:5000/kubesprayns/prom/prometheus:v2.2.1
$ sudo docker push  portus.oooooo:5000/kubesprayns/prom/pushgateway:v0.5.1
$ sudo docker push  portus.oooooo:5000/kubesprayns/prom/alertmanager:v0.14.0

項目CI/CD所需容器(test環節/build環節):

# docker pull golang:1.10.3-stretch
# docker tag golang:1.10.3-stretch portus.ooooooo.com:5000/kubesprayns/golang:1.10.3-stretch
# docker push portus.ooooooo.com:5000/kubesprayns/golang:1.10.3-stretch

Release階段所需容器:

# docker pull docker:latest
# docker pull lordgaav/dind-options:latest
# docker tag docker:latest portus.ooooooo.com:5000/kubesprayns/docker:latest
# docker push portus.ooooooo.com:5000/kubesprayns/docker:latest
# docker tag lordgaav/dind-options:latest portus.ooooooo.com:5000/lordgaav/dind-options:latest
# docker push portus.ooooooo.com:5000/kubesprayns/lordgaav/dind-options:latest

Deploy階段所需容器:

# docker pull lachlanevenson/k8s-kubectl:latest
# docker tag lachlanevenson/k8s-kubectl:latest portus.ooooooo.com:5000/kubesprayns/lachlanevenson/k8s-kubectl:latest
# docker push portus.ooooooo.com:5000/kubesprayns/lachlanevenson/k8s-kubectl:latest

Build 容器所需要的:

# docker pull busybox:1.28.4-glibc
# docker tag busybox:1.28.4-glibc portus.ooooooo.com:5000/kubesprayns/busybox:1.28.4-glibc
# docker push portus.ooooooo.com:5000/kubesprayns/busybox:1.28.4-glibc

離線charts準備

下載并更改charts, 而后上傳到私有倉庫,

gitlab-runner

下面是gitlab-runner的charts本地化制作過程:

$ helm repo add runner https://charts.gitlab.io
$ helm fetch runner/gitlab-runner
$ tar xzvf gitlab-runner-0.1.33.tgz
$ cd gitlab-runner
$ vim values.yaml
- gitlab/gitlab-runner:alpine-v10.3.0
+ image: portus.ooooooo.com:5000/kubesprayns/gitlab/gitlab-runner:alpine-v10.3.0

init:
-  image: busybox
+  image: portus.ooooooo.com:5000/kubesprayns/busybox

-  image: ubuntu:16.04
+  image: portus.ooooooo.com:5000/kubesprayns/ubuntu:16.04

-  helpers: {}
+  helpers:

-    # image: gitlab/gitlab-runner-helper:x86_64-latest
+    image: portus.ooooooo.com:5000/kubesprayns/gitlab/gitlab-runner-helper:x86_64-latest
$ helm package .
$ curl --data-binary "@gitlab-runner-0.1.33.tgz" http://portus.ooooooo.com:8988/api/charts

nginx-ingress離線charts

步驟:

# helm fetch stable/nginx-ingress
# tar xzvf nginx-ingress-0.28.2.tgz
# cd nginx-ingress
# vim values.yaml

<     repository: quay.io/kubernetes-ingress-controller/nginx-ingress-controller
---
>     repository: portus.ooooooo.com:5000/kubesprayns/quay.io/kubernetes-ingress-controller/nginx-ingress-controller
293c293
<     repository: k8s.gcr.io/defaultbackend
---
>     repository: portus.ooooooo.com:5000/kubesprayns/k8s.gcr.io/defaultbackend
# helm package .
$ curl --data-binary "@nginx-ingress-0.28.2.tgz" http://portus.ooooooo.com:8988/api/charts

需要配置默認的nginx-ingress-controller的配置:

# vim /opt/gitlab/embedded/service/gitlab-rails/vendor/ingress/values.yaml
- repository: "quay.io/kubernetes-ingress-controller/nginx-ingress-controller"
+ repository: "portus.oooooo.com:5000/kubesprayns/quay.io/kubernetes-ingress-controller/nginx-ingress-controller"
# gitlab-ctl reconfigure && gitlab-ctl restart

Prometheus離線charts

步驟:

$ helm fetch stable/prometheus --version 6.7.3
$ tar xzvf prometheus-6.7.3.tgz
$ cd prometheus/
$ vim values
35c35
<     repository: prom/alertmanager
---
>     repository: portus.oooooo.com:5000/kubesprayns/prom/alertmanager
212c212
<     repository: jimmidyson/configmap-reload
---
>     repository: portus.oooooo.com:5000/kubesprayns/jimmidyson/configmap-reload
247c247
<     repository: busybox
---
>     repository: portus.oooooo.com:5000/kubesprayns/busybox
268c268
<     repository: quay.io/coreos/kube-state-metrics
---
>     repository: portus.oooooo.com:5000/kubesprayns/quay.io/coreos/kube-state-metrics
345c345
<     repository: prom/node-exporter
---
>     repository: portus.oooooo.com:5000/kubesprayns/prom/node-exporter
439c439
<     repository: prom/prometheus
---
>     repository: portus.oooooo.com:5000/kubesprayns/prom/prometheus
653c653
<     repository: prom/pushgateway
---
>     repository: portus.oooooo.com:5000/kubesprayns/prom/pushgateway
$ helm package .
$ curl --data-binary "@prometheus-6.7.3.tgz" http://portus.ooooooo.com:8988/api/charts

測試項目準備

測試項目來自于github上的項目:

$ git clone https://github.com/galexrt/presentation-gitlab-k8s.git

搭建/配置GitLab

本文寫作時用到的GitLab版本為GitLab Community Edition 11.0.3 aa62075, 供參考.

# dpkg -i gitlab-ce_11.0.3-ce.0_amd64.deb
# dpkg -i gitlab-runner_11.0.2_amd64.deb

內網環境下為了避免網絡問題,關閉ufw防火墻:

# ufw disable

(可選)GitLab默認使用UTF-8,如果配置時碰到編碼問題,可在配置完本地編碼后,重新登錄終端執行reconfigure.

# cat > /etc/default/locale <<EOF
LC_ALL=en_US.UTF-8
LANG=en_US.UTF-8
LANGUAGE=en_US.UTF-8
EOF
# localedef -v -c -i en_US -f UTF-8 en_US.UTF-8 || true

配置gitlab監聽的IP端口, 并執行reconfigure, 此后gitlab服務將變得可用:

# vim /etc/gitlab/gitlab.rb
external_url 'http://192.168.122.160'
# gitlab-ctl reconfigure && gitlab-ctl restart

配置完成后設置用戶名/密碼, 搭建成功后的GitLab服務如下:

http://www.sqtpkw.live/wp-content/uploads/images/201901/201901021641163a0d0.jpeg
導入項目:

http://www.sqtpkw.live/wp-content/uploads/images/201901/20190102164117cd064.jpeg

在準備好的項目中,添加剛才在 Gitlab中創建的項目并提交:

# git remote remove origin
# git remote add origin [email protected]:root/testk8sci.git
# git push origin master

此時可以看到,代碼已經被提交到GitLab倉庫中,然而ci/cd處于pending狀態,接下來將對代碼進行修改,將其一步步調試通。

http://www.sqtpkw.live/wp-content/uploads/images/201901/201901021641192f246.jpeg

Gitlab Runner配置

首先需要添加kubernetes集群,而后在集群上安裝Helm Tiller, 安裝完Helm
Tiller后才可以繼續安裝GitLab Runner。 選取的項目需要使用GitLab
Runner才可以跑通CI/CD.

添加Kubernetes集群

點擊項目-> Operations->Kubernetes, 如下圖:

http://www.sqtpkw.live/wp-content/uploads/images/201901/2019010216412071388.jpeg

點擊Add Kubernetes Cluster按鈕,添加一個新的Kubernetes集群:

http://www.sqtpkw.live/wp-content/uploads/images/201901/201901021641213bc26.jpeg

此時有兩個選項,因為是離線的Kubernetes集群,點擊Add an existing Kubernetes
cluster
,繼續進入到下一步:

http://www.sqtpkw.live/wp-content/uploads/images/201901/20190102164124ce3a9.jpeg

事先需要獲得Kubernetes集群的管理信息,通過以下步驟獲得,

獲得API Server的地址(runner1節點上):

# kubectl config view | grep server
	server: https://192.168.122.161:6443

獲得CA證書:

# cat ~/.kube/config | grep certificate-authority-data: 
    certificate-authority-data: XXXXXXXXXXXXXXXXXXXXXXXXXXXX
# cat xxxxxxxxxxxxxxxxxxxxxxxxxx | base64 -d
-----BEGIN CERTIFICATE-----
xxxxxxxxx
-----END CERTIFICATE-----

獲得Token:

# kubectl get secret
NAME                  TYPE                                  DATA      AGE
default-token-spz8w   kubernetes.io/service-account-token   3         58m
# kubectl get secret default-token-spz8w -o yaml | grep token:
xxxxxxxxxxxx
# cat xxxxxxx | base64 -d

將上面獲得的各個項目填入到配置項目中:

http://www.sqtpkw.live/wp-content/uploads/images/201901/20190102164127d8663.jpeg

安裝Helm Tiller

安裝界面如下:

http://www.sqtpkw.live/wp-content/uploads/images/201901/20190102164128bcd5e.jpeg

直接點擊Install按鈕, 出現的問題為:

Kubernetes error: namespaces "gitlab-managed-apps" is forbidden: User
"system:serviceaccount:default:default" cannot get namespaces in the namespace
"gitlab-managed-apps"

http://www.sqtpkw.live/wp-content/uploads/images/201901/20190102164128f4dce.jpeg

搜索網上的解決方案,

# kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --user=admin --user=kubelet --group=system:serviceaccounts

按F5刷新頁面并點擊Install重新安裝,安裝會一直處于pause,最終將失敗,更改以下配置以便繼續安裝:

http://www.sqtpkw.live/wp-content/uploads/images/201901/2019010216413014be5.jpeg

# vim /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/kubernetes/helm/pod.rb +28
-            image: 'alpine:3.6',
+            image: 'portus.ooooooo.com:5000/kubesprayns/alpinewithwget:3.6',
# vim /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/kubernetes/helm/base_command.rb
-            ALPINE_VERSION=$(cat /etc/alpine-release | cut -d '.' -f 1,2)
-            echo http://mirror.clarkson.edu/alpine/v$ALPINE_VERSION/main >> /etc/apk/repositories
-            echo http://mirror1.hs-esslingen.de/pub/Mirrors/alpine/v$ALPINE_VERSION/main >> /etc/apk/repositories
-            apk add -U wget ca-certificates openssl >/dev/null
-            wget -q -O - https://kubernetes-helm.storage.googleapis.com/helm-v#{Gitlab::Kubernetes::Helm::HELM_VERSION}-linux-amd64.tar.gz | tar zxC /tmp >/dev/null
+            wget -q -O - http://portus.ooooooo.com:8888/helm-v2.7.0-linux-amd64.tar.gz | tar zxC /tmp >/dev/null

更改完畢后,我們重新配置gitlab以繼續安裝:

# gitlab-ctl reconfigure && gitlab-ctl restart

此時會出現helm在初始化的時候因為訪問不到外網出現的timeout錯誤,如下:

http://www.sqtpkw.live/wp-content/uploads/images/201901/20190102164131d4dcb.jpeg
更改helm init的初始化方式:

# vim /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/kubernetes/helm/install_command.rb
-          'helm init --client-only >/dev/null'
+          'helm init --stable-repo-url http://portus.ooooooo.com:8988 --tiller-image portus.ooooooo.com:5000/kubesprayns/gcr.io/kubernetes-helm/tiller:v2.7.0 >/dev/null'
# /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/kubernetes/helm/init_command.rb
-          "helm init >/dev/null"
+          'helm init --stable-repo-url http://portus.ooooooo.com:8988 --tiller-image portus.ooooooo.com:5000/kubesprayns/gcr.io/kubernetes-helm/tiller:v2.7.0 >/dev/null'

更改完畢后,我們重新配置gitlab以繼續安裝:

# gitlab-ctl reconfigure && gitlab-ctl restart

此時終于可以安裝成功helm:

http://www.sqtpkw.live/wp-content/uploads/images/201901/20190102164132dcc5d.jpeg

gitlab-runner

默認點擊安裝將出現以下錯誤,:

http://www.sqtpkw.live/wp-content/uploads/images/201901/2019010216413515390.jpeg

$ vim /opt/gitlab/embedded/service/gitlab-rails/app/models/clusters/applications/runner.rb
-        'https://charts.gitlab.io'
+        'http://portus.ooooooo.com:8988'

再次運行,提示configmap已經存在:

http://www.sqtpkw.live/wp-content/uploads/images/201901/20190102164136ca38a.jpeg

刪除configmap, 繼續運行:

# kubectl delete configmap values-content-configuration-runner -n gitlab-managed-apps

至此,我們的gitlab-runner也安裝成功:

http://www.sqtpkw.live/wp-content/uploads/images/201901/201901021641361428c.jpeg

Ingress

默認情況下安裝將觸發以下錯誤:

http://www.sqtpkw.live/wp-content/uploads/images/201901/2019010216413720610.jpeg
上傳完對應的charts文件以后,繼續安裝:

http://www.sqtpkw.live/wp-content/uploads/images/201901/20190102164138d81f7.jpeg

刪除configmaps:

# kubectl delete configmap values-content-configuration-ingress -n gitlab-managed-apps

安裝成功后:

http://www.sqtpkw.live/wp-content/uploads/images/201901/2019010216413920222.jpeg

Prometheus

與上述步驟相似,安裝完畢后:

http://www.sqtpkw.live/wp-content/uploads/images/201901/201901021641394d022.jpeg

值得注意的是,需要有內建的默認存儲提供,否則pod將一直處于Initialize狀態。

CI/CD項目

此時提交已經可以觸發runner,

http://www.sqtpkw.live/wp-content/uploads/images/201901/2019010216414074d73.jpeg

針對不同的Stage進行Debug. 最終跑通編譯。

Prometheus

開啟前,集群需要有內建的默認存儲,例如:

# kubectl get sc 
NAME                    PROVISIONER      AGE
nfs-storage (default)   fuseim.pri/ifs   35m

未開啟前:

http://www.sqtpkw.live/wp-content/uploads/images/201901/20190102164141104e6.jpeg

開啟步驟:

$ vim /etc/gitlab/gitlab.rb
################################################################################
## Prometheus
##! Docs: https://docs.gitlab.com/ce/administration/monitoring/prometheus/
################################################################################

prometheus['enable'] = true
prometheus['monitor_kubernetes'] = true
prometheus['listen_address'] = "0.0.0.0:9090"
prometheus['username'] = 'gitlab-prometheus'
prometheus['uid'] = nil
prometheus['gid'] = nil
prometheus['shell'] = '/bin/sh'
prometheus['home'] = '/var/opt/gitlab/prometheus'
prometheus['log_directory'] = '/var/log/gitlab/prometheus'
prometheus['scrape_interval'] = 15
prometheus['scrape_timeout'] = 15
prometheus['chunk_encoding_version'] = 2

# To completely disable prometheus, and all of it's exporters, set to false
prometheus_monitoring['enable'] = true

$ gitlab-ctl reconfigure && gitlab-ctl restart

同時,需要更改項目中的deployment的名稱:

$ vim manifests/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
-  name: presentation-gitlab-k8s-__CI_ENVIRONMENT_SLUG__
+  name: __CI_ENVIRONMENT_SLUG__-presentation-gitlab-k8s

這是因為:

http://www.sqtpkw.live/wp-content/uploads/images/201901/201901021641440e8a1.jpeg

注意到Kubernetes cluster details的Project namespace環節要設置為空:

http://www.sqtpkw.live/wp-content/uploads/images/201901/20190102164145f4f2b.jpeg
現在點擊Options->Environments, 查看其監控指標:

http://www.sqtpkw.live/wp-content/uploads/images/201901/20190102164149ee96c.jpeg

七星彩走势图2元网官网