k8s

K8S集群安装(2)-Harbor镜像仓库安装

Harbor是一个私有的镜像仓库，可以用于存放Docker镜像，在K8S当中部署一个项目需要从镜像仓库拉取镜像，再去构建Pod。 为什么我们要去部署属于自己的私有镜像仓库Harbor？ 1.K8S集群当中进行项目的构建时，如果从DockerHub进行镜像的拉取，因为墙的原因很容易遇到拉取不到镜像的情

Harbor是一个私有的镜像仓库，可以用于存放Docker镜像，在K8S当中部署一个项目需要从镜像仓库拉取镜像，再去构建Pod。

为什么我们要去部署属于自己的私有镜像仓库Harbor？

  • 1.K8S集群当中进行项目的构建时，如果从DockerHub进行镜像的拉取，因为墙的原因很容易遇到拉取不到镜像的情况，此时如果我们有私有镜像的话，我们可以将镜像上传到私有仓库，K8S项目的构建(例如Deployment)，就可以从私有镜像仓库Harbor进行镜像拉取。
  • 2.当我们自己的项目发布时，需要构建自己的Docker镜像，并push到远程仓库，再通过K8S拉取镜像再进行服务的构建，此时Harbor就发挥到它作为镜像仓库的作用。

下面，我们将会使用虚拟机基于Harbor的离线安装包，并基于docker-compose去部署Harbor私有镜像仓库。

1.独立机器部署Harbor私服镜像仓库

1.1 安装本地的Harbor镜像仓库

Harbor安装，下载地址如下：https://github.com/goharbor/harbor/releases/。

habor-release-page-select.png

上面是Github页面，可以选择右边的harbor-online-installer-v1.10.19.tgz在线版本或者是harbor-offline-installer-v1.10.19.tgz离线版本进行安装。这里推荐使用offline离线版本进行安装，因为Harbor在线版本需要在线下载依赖可能遇到网速问题，离线版本是包当中已经带了所有的Harbor相关的依赖。

下载完成Harbor的离线版本的压缩包之后，我们通过解压并执行安装脚本的方式去安装Harbor。

tar -xvzf harbor-offline-installer-*.tgz

进入到harbor/目录下，可以看到Harbor的主配置文件harbor.yml，我们可能需要进行一些自定义的配置才能启动Harbor。

# Configuration file of Harbor

# The IP address or hostname to access admin UI and registry service.
# DO NOT use localhost or 127.0.0.1, because Harbor needs to be accessed by external clients.
hostname: xxx.xxx.com

# http related config
http:
  # port for http, default is 80. If https enabled, this port will redirect to https port
  port: 84

# https related config
# https:
  # https port for harbor, default is 443
  # port: 444
  # The path of cert and key files for nginx
  # certificate: /your/certificate/path
  # private_key: /your/private/key/path

# Uncomment external_url if you want to enable external proxy
# And when it enabled the hostname will no longer used
# external_url: https://reg.mydomain.com:8433

# The initial password of Harbor admin
# It only works in first time to install harbor
# Remember Change the admin password from UI after launching Harbor.
harbor_admin_password: Harbor12345

# Harbor DB configuration
database:
  # The password for the root user of Harbor DB. Change this before any production use.
  password: root123
  # The maximum number of connections in the idle connection pool. If it <=0, no idle connections are retained.
  max_idle_conns: 50
  # The maximum number of open connections to the database. If it <= 0, then there is no limit on the number of open connections.
  # Note: the default number of connections is 100 for postgres.
  max_open_conns: 100

# The default data volume
data_volume: /data

# Harbor Storage settings by default is using /data dir on local filesystem
# Uncomment storage_service setting If you want to using external storage
# storage_service:
#   # ca_bundle is the path to the custom root ca certificate, which will be injected into the truststore
#   # of registry's and chart repository's containers.  This is usually needed when the user hosts a internal storage with self signed certificate.
#   ca_bundle:

#   # storage backend, default is filesystem, options include filesystem, azure, gcs, s3, swift and oss
#   # for more info about this configuration please refer https://docs.docker.com/registry/configuration/
#   filesystem:
#     maxthreads: 100
#   # set disable to true when you want to disable registry redirect
#   redirect:
#     disabled: false

# Clair configuration
clair:
  # The interval of clair updaters, the unit is hour, set to 0 to disable the updaters.
  updaters_interval: 12

jobservice:
  # Maximum number of job workers in job service
  max_job_workers: 10

notification:
  # Maximum retry count for webhook job
  webhook_job_max_retry: 10

chart:
  # Change the value of absolute_url to enabled can enable absolute url in chart
  absolute_url: disabled

# Log configurations
log:
  # options are debug, info, warning, error, fatal
  level: info
  # configs for logs in local storage
  local:
    # Log files are rotated log_rotate_count times before being removed. If count is 0, old versions are removed rather than rotated.
    rotate_count: 50
    # Log files are rotated only if they grow bigger than log_rotate_size bytes. If size is followed by k, the size is assumed to be in kilobytes.
    # If the M is used, the size is in megabytes, and if G is used, the size is in gigabytes. So size 100, size 100k, size 100M and size 100G
    # are all valid.
    rotate_size: 200M
    # The directory on your host that store log
    location: /var/log/harbor

  # Uncomment following lines to enable external syslog endpoint.
  # external_endpoint:
  #   # protocol used to transmit log to external endpoint, options is tcp or udp
  #   protocol: tcp
  #   # The host of external endpoint
  #   host: localhost
  #   # Port of external endpoint
  #   port: 5140

#This attribute is for migrator to detect the version of the .cfg file, DO NOT MODIFY!
_version: 1.10.0

# Uncomment external_database if using external database.
# external_database:
#   harbor:
#     host: harbor_db_host
#     port: harbor_db_port
#     db_name: harbor_db_name
#     username: harbor_db_username
#     password: harbor_db_password
#     ssl_mode: disable
#     max_idle_conns: 2
#     max_open_conns: 0
#   clair:
#     host: clair_db_host
#     port: clair_db_port
#     db_name: clair_db_name
#     username: clair_db_username
#     password: clair_db_password
#     ssl_mode: disable
#   notary_signer:
#     host: notary_signer_db_host
#     port: notary_signer_db_port
#     db_name: notary_signer_db_name
#     username: notary_signer_db_username
#     password: notary_signer_db_password
#     ssl_mode: disable
#   notary_server:
#     host: notary_server_db_host
#     port: notary_server_db_port
#     db_name: notary_server_db_name
#     username: notary_server_db_username
#     password: notary_server_db_password
#     ssl_mode: disable

# Uncomment external_redis if using external Redis server
# external_redis:
#   host: redis
#   port: 6379
#   password:
#   # db_index 0 is for core, it's unchangeable
#   registry_db_index: 1
#   jobservice_db_index: 2
#   chartmuseum_db_index: 3
#   clair_db_index: 4

# Uncomment uaa for trusting the certificate of uaa instance that is hosted via self-signed cert.
# uaa:
#   ca_file: /path/to/ca

# Global proxy
# Config http proxy for components, e.g. http://my.proxy.com:3128
# Components doesn't need to connect to each others via http proxy.
# Remove component from `components` array if want disable proxy
# for it. If you want use proxy for replication, MUST enable proxy
# for core and jobservice, and set `http_proxy` and `https_proxy`.
# Add domain to the `no_proxy` field, when you want disable proxy
# for some special registry.
proxy:
  http_proxy:
  https_proxy:
  # no_proxy endpoints will appended to 127.0.0.1,localhost,.local,.internal,log,db,redis,nginx,core,portal,postgresql,jobservice,registry,registryctl,clair,chartmuseum,notary-server
  no_proxy:
  components:
    - core
    - jobservice
    - clair

在Harbor配置当中，我们需要配置Harbor的HTTP和HTTPS的相关配置：

  • 在hostname出填写域名配置（如果配置了反向代理则不需要）
  • 在http.port当中配置HTTP服务启动的端口号。
  • (选填，在需要使用Harbor的HTTPS时才需要配置)在https.port当中配置HTTPS服务启动的端口号，https.certificate当中配置HTTPS的证书所在的地址，https.private_key当中配置HTTPS的私钥所在的地址。
hostname: xxx.xxx.com

http:
  port: 84
https:
  port: 1443
  certificate: /your/certificate/path
  private_key: /your/private/key/path

在配置好Harbor配置文件之后，我们通过如下的命令安装脚本install，去安装好Harbor，并启动Harbor。

./install.sh

如果想要启动/关闭Harbor，我们可以通过如下的命令去进行操作，

# 启动habor
docker-compose up -d
# 关闭habor
docker-compose down

接着通过docker ps可以查看运行的容器情况，在这里这里可以找到Harbor的相关的运行的容器。

root@ecm-4fe6:~/tools/habor/harbor# docker ps
CONTAINER ID   IMAGE                                  COMMAND                  CREATED              STATUS                        PORTS                                     NAMES
fc68663a0c75   goharbor/harbor-jobservice:v1.10.19    "/harbor/harbor_jobs…"   About a minute ago   Up 58 seconds (healthy)                                                 harbor-jobservice
294528f398de   goharbor/nginx-photon:v1.10.19         "nginx -g 'daemon of…"   About a minute ago   Up 58 seconds (healthy)       0.0.0.0:84->8080/tcp, [::]:84->8080/tcp   nginx
43b3bcdf06cb   goharbor/harbor-core:v1.10.19          "/harbor/harbor_core"    About a minute ago   Up About a minute (healthy)                                             harbor-core
1a7ae4231f1c   goharbor/harbor-portal:v1.10.19        "nginx -g 'daemon of…"   About a minute ago   Up About a minute (healthy)   8080/tcp                                  harbor-portal
9078844124c4   goharbor/harbor-db:v1.10.19            "/docker-entrypoint.…"   About a minute ago   Up About a minute (healthy)   5432/tcp                                  harbor-db
72f314603d16   goharbor/registry-photon:v1.10.19      "/home/harbor/entryp…"   About a minute ago   Up About a minute (healthy)   5000/tcp                                  registry
76680fc53a23   goharbor/harbor-registryctl:v1.10.19   "/home/harbor/start.…"   About a minute ago   Up About a minute (healthy)                                             registryctl
253c363e7abd   goharbor/redis-photon:v1.10.19         "redis-server /etc/r…"   About a minute ago   Up About a minute (healthy)   6379/tcp                                  redis
643df5ccdcb1   goharbor/harbor-log:v1.10.19           "/bin/sh -c /usr/loc…"   About a minute ago   Up About a minute (healthy)   127.0.0.1:1514->10514/tcp                 harbor-log

Harbor的HTTP服务，在84端口启动，访问{ip}:84进入Harbor管理如下的Harbor后台页面。

harbor-login-page.png

接着使用账号密码登录，账号admin，默认密码在harbor.yml配置文件当中，首次登录需要修改Harbor密码。

1.2 通过已有Nginx服务反向代理Harbor服务请求(可选)

对于Harbor的SSL配置，如果在已有Nginx服务当中配置有HTTPS的SSL证书，我们也可以使用反向代理，将Harbor的84端口通过Nginx进行反向代理暴露到外部，我们添加如下的配置去进行反向代理。编辑Nginx服务器配置，使用命令如下：sudo vim /etc/nginx/sites-available/default。

server {
        listen 188 default_server;
        listen [::]:188 default_server;

        # SSL configuration
        listen 8443 ssl default_server;
        listen [::]:8443 ssl default_server;
        ssl_certificate  /root/ssl/wanna1314y.top_bundle.crt;
        # 私钥文件的相对路径或绝对路径
        ssl_certificate_key /root/ssl/wanna1314y.top.key;
        ssl_session_timeout 5m;
        # 配置加密套件，写法遵循 openssl 标准。
        ssl_ciphers ECDHE-RSA-AES128-GCM-SHA256:ECDHE:ECDH:AES:HIGH:!NULL:!aNULL:!MD5:!ADH:!RC4;
        # 协议配置
        ssl_protocols TLSv1.2 TLSv1.3;
        ssl_prefer_server_ciphers on;

        root /var/www/html;

        # Add index.php to the list if you are using PHP
        index index.html index.htm index.nginx-debian.html;

        server_name wanna1314y.top;

        location / {
                proxy_pass http://127.0.0.1:84;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
        }
}

我们想要将Nginx通过1443端口对外HTTPS暴露，代理从而内部的http://127.0.0.1:84接口(Harbor服务)。(需要修改ssl_certificate和ssl_certificate_key，指定SSL证书所在的位置)

通过如下这样的配置，在防火墙(安全组)当中放开1443端口，接着就可以访问我们的1443端口基于HTTPS协议的方式去访问Harbor啦，地址参考：https://wanna1314y.top:1443。

1.3 将本地Docker镜像推送到Harbor仓库

我们下面以构建并推送一个nginx镜像举例。我们首先需要拉取(或者构建一个镜像)

docker pull nginx

我们通过docker images可以查看本地仓库都有哪些镜像，比如我们上面pull了一个nginx镜像，那么这里就可以看到之前我们拉取到的nginx镜像。

root@ecm-4fe6:~# docker images
REPOSITORY                      TAG        IMAGE ID       CREATED       SIZE
nginx                           latest     f02a7f566928   3 weeks ago   117MB
goharbor/chartmuseum-photon     v1.10.19   222e01a0c4f8   6 weeks ago   177MB
goharbor/redis-photon           v1.10.19   8abbe7c628db   6 weeks ago   148MB
goharbor/clair-adapter-photon   v1.10.19   1f856e990872   6 weeks ago   81.9MB
goharbor/clair-photon           v1.10.19   0ccce83decfa   6 weeks ago   181MB
goharbor/notary-server-photon   v1.10.19   fe7aae6b169d   6 weeks ago   119MB
goharbor/notary-signer-photon   v1.10.19   ca2cfca2baf4   6 weeks ago   116MB
goharbor/harbor-registryctl     v1.10.19   fc4186073195   6 weeks ago   109MB
goharbor/registry-photon        v1.10.19   b5dc0c94b21e   6 weeks ago   92.1MB
goharbor/nginx-photon           v1.10.19   4b6a3a04832e   6 weeks ago   122MB
goharbor/harbor-log             v1.10.19   08fa03691200   6 weeks ago   133MB
goharbor/harbor-jobservice      v1.10.19   1b5aca7f4143   6 weeks ago   97.8MB
goharbor/harbor-core            v1.10.19   85627df50bd9   6 weeks ago   91.6MB
goharbor/harbor-portal          v1.10.19   e72ed626ae05   6 weeks ago   130MB
goharbor/harbor-db              v1.10.19   c038e2ac8cb3   6 weeks ago   213MB
goharbor/prepare                v1.10.19   3abb30b0d645   6 weeks ago   170MB

接着我们需要给镜像打TAG。

# 完整命令格式
# 这里的{repository}/{project}是该镜像在本地的镜像名称，{tag}就是该依赖在本地的版本(默认是latest)
# 这里的{remoteRepository}/{remoteProject}对应的是远程仓库的地址和远程仓库的项目名称, {imageName}是镜像名称(比如nginx)，{remoteTag}则是需要推送到远程的版本(默认是latest)
docker tag {repository}/{project}:{tag} {remoteRepository}/{remoteProject}/{imageName}:{remoteTag}


# 我们这里需要将本地的nginx推送到远程仓库(127.0.0.1:84)，远程仓库的项目是library(Harbor仓库默认就是library)，
# 本地镜像tag为nginx:latest, 远程的镜像tag是127.0.0.1:84/library/nginx:latest
docker tag nginx:latest 127.0.0.1:84/library/nginx:latest
# 如果是版本是latest, 那么可以省略掉, 例如上面的命令可以简写成为下面的
docker tag nginx 127.0.0.1:84/library/nginx

接着使用docker push尝试推送到本地私服

docker push 127.0.0.1:84/library/nginx

docker push执行结果如下：

Using default tag: latest
The push refers to repository [127.0.0.1:84/library/nginx]
c13ac49c8f3c: Preparing 
5f70bf18a086: Preparing 
813c9361c9a1: Preparing 
352667057690: Preparing 
9f29839ee958: Preparing 
8895fa2456bd: Waiting 
2ceb45dcc459: Waiting 
98b5f35ea9d3: Waiting 
denied: requested access to the resource is denied

推送过程中，发现denied: requested access to the resource is denied，原因在于我们没有权限，我们需要首先使用docker login 127.0.0.1:84命令，先登录Harbor账号。

Username: admin
Password: 
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credential-stores

Login Succeeded

中间过程会让提示用户名和密码，看到Login Succeeded之后，我们重新推送镜像。

root@ecm-4fe6:~# docker push 127.0.0.1:84/library/nginx
Using default tag: latest
The push refers to repository [127.0.0.1:84/library/nginx]
c13ac49c8f3c: Pushed 
5f70bf18a086: Pushed 
813c9361c9a1: Pushed 
352667057690: Pushed 
9f29839ee958: Pushed 
8895fa2456bd: Pushed 
2ceb45dcc459: Pushed 
98b5f35ea9d3: Pushed 
latest: digest: sha256:6df8098f056f25e26890a11bbe06fa59a6fd035691d1b2747ff192e1a04b4e0b size: 1986

看到上述的提示说明我们已经成功推送到本地Harbor私服。

2.Harbor相关的配置信息

2.1 修改Harbor的数据存放路径

如果Harbor现在已经有了数据，那么如果需要去进行数据迁移，那么需要先将原来的数据进行迁移，Harbor默认将数据存放到/data目录下，我们现在将/data目录下的内容拷贝到/sharedata/data目录下，使用如下的命令拷贝：

# 安装rsync工具
sudo apt install rsync

# 数据拷贝
rsync -av /data /sharedata

接着找到Harbor的安装目录下的harbor.yaml配置文件，修改data_valume配置，将数据卷目录去配置为/sharedata/data。

# 修改前
data_volume: /data
# 修改后
data_volume: /sharedata/data

接着找到Harbor安装目录下的安装脚本，使用./install.sh执行Harbor的重新安装，期间会停止之前已经启动的Harbor，重新生成docker-compose.yaml文件。

需要注意的是：执行install.sh这里会重新生成docker-compose.yaml文件，因此会修改所有的数据卷的路径配置！！！

2.2 修改Harbor的外部访问地址(Nginx反向代理Harbor时需要)

在配置了反向代理访问Harbor的情况，需要配置Harbor的外部访问地址。

如果我们通过Nginx反向代理去访问Harbor，那么很可能出现Harbor的启动端口号和Nginx的启动端口号不一致的情况。比如我们的Nginx服务在443端口启动，并且在Nginx上配置了一个域名aaa.xxx.com，但是Harbor仓库在84端口启动，通过Nginx反向代理Harbor的84端口。那么如果Harbor内部存在有页面跳转等情况，Harbor可能会跳转到aaa.xxx.com:84端口，因为对Harbor来说，其实外部访问它的就是84端口，但是aaa.xxx.com:84很可能是一个内部使用的接口(或者Harbor本身是部署在其他的机器上的，根本就不存在这样的接口)，对用户来说是无法访问的，用户感知的是443端口，直接进行页面跳转时却跳转到aaa.xxx.com:84端口，属于是页面的错误跳转。

因此如果我们通过Nginx反向代理的Harbor，修改外部访问地址是很有必要的，例如我们对外通过Nginx访问Harbor的接口是https://aaa.xxx.con:443，那么我们就需要配置Harbor的外部访问地址是https://aaa.xxx.con:443。

我们找到harbor.yml配置文件，新增如下的配置项：

external_url: https://aaa.xxx.com:443

如果不修改外部访问地址的话，最典型的影响我们使用的就是Harbor页面上复制的docker pull命令是不对的。

# 不配置external_url
docker pull aaa.xxx.com:84/library/redis:latest

# 配置external_url之后, 自动修正
docker pull aaa.xxx.com:443/library/redis:latest

修改harbor.yml配置文件之后，我们使用./install.sh命令去重启Harbor服务，实现重新加载Harbor的配置文件。

Comment