[关闭]
@zhangyy 2021-07-16T10:30:44.000000Z 字数 24178 阅读 593

kubeadmin 安装k8s1.20集群

kubernetes架构系列

一：k8s1.20.x 的重要更新

  1. 1、Kubectl debug 设置一个临时容器
  2. 2、Sidecar 
  3. 3、Volume：更改目录权限，fsGroup
  4. 4、ConfigMap和Secret
  6. K8S官网：https://kubernetes.io/docs/setup/
  7. 最新版高可用安装：https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/

二：k8s1.20.x 的安装

2.1 ：高可用Kubernetes集群规划

image_1f0sgr8qq1d2inqr35ml3g1qsu9.png-43.5kB

  1. k8s 的高可用的架构图

image_1f1m9lqt31do01d3t1rgejtbfnn9.png-131.5kB

  1. 所有节点配置hosts，修改/etc/hosts如下：
  2. cat /etc/hosts 
  3. ----
  4. 192.168.100.11  node01.flyfish.cn
  5. 192.168.100.12  node02.flyfish.cn
  6. 192.168.100.13  node03.flyfish.cn
  7. 192.168.100.14  node04.flyfish.cn
  8. 192.168.100.15  node05.flyfish.cn
  9. 192.168.100.16  node06.flyfish.cn
  10. 192.168.100.17  node07.flyfish.cn
  11. 192.168.100.18  node08.flyfish.cn
  12. ----

2.2 yum 的更新配置 （所有节点全部安装）

  1. curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
  2. yum install -y yum-utils device-mapper-persistent-data lvm2
  3. yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
  4. cat <<EOF > /etc/yum.repos.d/kubernetes.repo
  5. [kubernetes]
  6. name=Kubernetes
  7. baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
  8. enabled=1
  9. gpgcheck=1
  10. repo_gpgcheck=1
  11. gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
  12. EOF
  13. sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo

image_1f0shlio9u63h1ct111vne154m.png-246.8kB

  1. 必备工具安装:
  2.       yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 git -y

image_1f0shp7p44mc1vit1cf1grd1i2s13.png-120.7kB

  1. 所有节点关闭防火墙、selinux、dnsmasq、swap。服务器配置如下：
  3. systemctl disable --now firewalld 
  4. systemctl disable --now dnsmasq
  5. systemctl disable --now NetworkManager
  7. setenforce 0
  8. sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/sysconfig/selinux
  9. sed -i 's#SELINUX=enforcing#SELINUX=disabled#g' /etc/selinux/config
  1. 关闭swap分区 (全部节点)
  2.      swapoff -a && sysctl -w vm.swappiness=0
  3.      sed -ri '/^[^#]*swap/s@^@#@' /etc/fstab

image_1f0sht5fipl21jlj9ck1ljlt8a1g.png-58.1kB

  1. 安装ntpdate
  3. rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm
  4. yum install ntpdate -y
  5. 所有节点同步时间。时间同步配置如下：
  7. ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
  8. echo 'Asia/Shanghai' >/etc/timezone
  9. ntpdate time2.aliyun.com
  10. 加入到crontab
  12. */5 * * * * ntpdate time2.aliyun.com
  13. 所有节点配置limit：
  15. ulimit -SHn 65535
  17. vim /etc/security/limits.conf
  18. # 末尾添加如下内容
  19. * soft nofile 655360
  20. * hard nofile 131072
  21. * soft nproc 655350
  22. * hard nproc 655350
  23. * soft memlock unlimited
  24. * hard memlock unlimited
  1. 安装ntpdate
  3. rpm -ivh http://mirrors.wlnmp.com/centos/wlnmp-release-centos.noarch.rpm
  4. yum install ntpdate -y
  5. 所有节点同步时间。时间同步配置如下：
  7. ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
  8. echo 'Asia/Shanghai' >/etc/timezone
  9. ntpdate time2.aliyun.com
  10. 加入到crontab
  12. */5 * * * * ntpdate time2.aliyun.com
  13. 所有节点配置limit：
  15. ulimit -SHn 65535
  17. vim /etc/security/limits.conf
  18. # 末尾添加如下内容
  19. * soft nofile 655360
  20. * hard nofile 131072
  21. * soft nproc 655350
  22. * hard nproc 655350
  23. * soft memlock unlimited
  24. * hard memlock unlimited
  25. Master01节点免密钥登录其他节点：
  27. ssh-keygen -t rsa
  28. for i in k8s-master01.flyfish.cn k8s-master02.flyfish.cn k8s-master03.flyfish.cn k8s-node01.flyfish.cn k8s-node02.flyfish.cn;do ssh-copy-id -i .ssh/id_rsa.pub $i;done
  30. 所有节点升级系统并重启：
  32. yum update -y  && reboot
  1. 下载安装源码文件：
  2.     cd /root/ ; git clone https://github.com/dotbalo/k8s-ha-install.git
  1. CentOS 7安装yum源如下：
  2.         curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
  3. yum install -y yum-utils device-mapper-persistent-data lvm2
  4. yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
  5. cat <<EOF > /etc/yum.repos.d/kubernetes.repo
  6. [kubernetes]
  7. name=Kubernetes
  8. baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
  9. enabled=1
  10. gpgcheck=1
  11. repo_gpgcheck=1
  12. gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
  13. EOF
  14. sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo
  1. CentOS 8 安装源如下：
  2. curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-8.repo
  4. yum install -y yum-utils device-mapper-persistent-data lvm2
  5. yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo
  7. cat <<EOF > /etc/yum.repos.d/kubernetes.repo
  8. [kubernetes]
  9. name=Kubernetes
  10. baseurl=https://mirrors.aliyun.com/kubernetes/yum/repos/kubernetes-el7-x86_64/
  11. enabled=1
  12. gpgcheck=1
  13. repo_gpgcheck=1
  14. gpgkey=https://mirrors.aliyun.com/kubernetes/yum/doc/yum-key.gpg https://mirrors.aliyun.com/kubernetes/yum/doc/rpm-package-key.gpg
  15. EOF
  16. sed -i -e '/mirrors.cloud.aliyuncs.com/d' -e '/mirrors.aliyuncs.com/d' /etc/yum.repos.d/CentOS-Base.repo
  1. 所有节点升级系统并重启，此处升级没有升级内核，下节会单独升级内核：
  2. yum install wget jq psmisc vim net-tools telnet yum-utils device-mapper-persistent-data lvm2 -y
  3. yum update -y --exclude=kernel* && reboot #CentOS7需要升级，8不需要
  1. 1.1.2  内核配置
  2. CentOS7 需要升级内核至4.18+
  3. https://www.kernel.org/ 和 https://elrepo.org/linux/kernel/el7/x86_64/
  5. CentOS 7 dnf可能无法安装内核
  6. dnf --disablerepo=\* --enablerepo=elrepo -y install kernel-ml kernel-ml-devel
  7. grubby --default-kernel
  10. 使用如下方式安装最新版内核
  11. rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
  12. rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
  14. 查看最新版内核yum --disablerepo="*" --enablerepo="elrepo-kernel" list available
  16. [root@k8s-node01 ~]# yum --disablerepo="*" --enablerepo="elrepo-kernel" list available
  17. Loaded plugins: fastestmirror
  18. Loading mirror speeds from cached hostfile
  19.  * elrepo-kernel: mirrors.neusoft.edu.cn
  20. elrepo-kernel                                                                                  | 2.9 kB  00:00:00     
  21. elrepo-kernel/primary_db                                                                       | 1.9 MB  00:00:00     
  22. Available Packages
  23. elrepo-release.noarch                                      7.0-5.el7.elrepo                              elrepo-kernel
  24. kernel-lt.x86_64                                           4.4.229-1.el7.elrepo                          elrepo-kernel
  25. kernel-lt-devel.x86_64                                     4.4.229-1.el7.elrepo                          elrepo-kernel
  26. kernel-lt-doc.noarch                                       4.4.229-1.el7.elrepo                          elrepo-kernel
  27. kernel-lt-headers.x86_64                                   4.4.229-1.el7.elrepo                          elrepo-kernel
  28. kernel-lt-tools.x86_64                                     4.4.229-1.el7.elrepo                          elrepo-kernel
  29. kernel-lt-tools-libs.x86_64                                4.4.229-1.el7.elrepo                          elrepo-kernel
  30. kernel-lt-tools-libs-devel.x86_64                          4.4.229-1.el7.elrepo                          elrepo-kernel
  31. kernel-ml.x86_64                                           5.7.7-1.el7.elrepo                            elrepo-kernel
  32. kernel-ml-devel.x86_64                                     5.7.7-1.el7.elrepo                            elrepo-kernel
  33. kernel-ml-doc.noarch                                       5.7.7-1.el7.elrepo                            elrepo-kernel
  34. kernel-ml-headers.x86_64                                   5.7.7-1.el7.elrepo                            elrepo-kernel
  35. kernel-ml-tools.x86_64                                     5.7.7-1.el7.elrepo                            elrepo-kernel
  36. kernel-ml-tools-libs.x86_64                                5.7.7-1.el7.elrepo                            elrepo-kernel
  37. kernel-ml-tools-libs-devel.x86_64                          5.7.7-1.el7.elrepo                            elrepo-kernel
  38. perf.x86_64                                                5.7.7-1.el7.elrepo                            elrepo-kernel
  39. python-perf.x86_64                                         5.7.7-1.el7.elrepo                            elrepo-kernel
  41. 安装最新版：
  42. yum --enablerepo=elrepo-kernel install kernel-ml kernel-ml-devel –y
  43. 安装完成后reboot
  44. 更改内核顺序：
  45. grub2-set-default  0 && grub2-mkconfig -o /etc/grub2.cfg && grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)" && reboot
  46. 开机后查看内核
  47. [appadmin@k8s-node01 ~]$ uname -a
  48. Linux k8s-node01 5.7.7-1.el7.elrepo.x86_64 #1 SMP Wed Jul 1 11:53:16 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
  50. CentOS 8按需升级:
  51. 可以采用dnf升级，也可使用上述同样步骤升级（使用上述步骤注意elrepo-release-8.1版本）
  52. rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
  53. dnf install https://www.elrepo.org/elrepo-release-8.1-1.el8.elrepo.noarch.rpm
  56. dnf --disablerepo=\* --enablerepo=elrepo -y install kernel-ml kernel-ml-devel
  57. grubby --default-kernel && reboot
  1. 安装依赖包：
  2.     本所有节点安装ipvsadm：
  3. yum install ipvsadm ipset sysstat conntrack libseccomp -y
  4. 所有节点配置ipvs模块，在内核4.19+版本nf_conntrack_ipv4已经改为nf_conntrack，本例安装的内核为4.18，使用nf_conntrack_ipv4即可：
  5. modprobe -- ip_vs
  6. modprobe -- ip_vs_rr
  7. modprobe -- ip_vs_wrr
  8. modprobe -- ip_vs_sh
  9. modprobe -- nf_conntrack_ipv4
  10. cat /etc/modules-load.d/ipvs.conf 
  11. ip_vs
  12. ip_vs_rr
  13. ip_vs_wrr
  14. ip_vs_sh
  15. nf_conntrack_ipv4
  16. ip_tables
  17. ip_set
  18. xt_set
  19. ipt_set
  20. ipt_rpfilter
  21. ipt_REJECT
  22. ipip
  23. 然后执行systemctl enable --now systemd-modules-load.service即可
  1. 开启一些k8s集群中必须的内核参数，所有节点配置k8s内核：
  2.     cat <<EOF > /etc/sysctl.d/k8s.conf
  3. net.ipv4.ip_forward = 1
  4. net.bridge.bridge-nf-call-iptables = 1
  5. fs.may_detach_mounts = 1
  6. vm.overcommit_memory=1
  7. vm.panic_on_oom=0
  8. fs.inotify.max_user_watches=89100
  9. fs.file-max=52706963
  10. fs.nr_open=52706963
  11. net.netfilter.nf_conntrack_max=2310720
  13. net.ipv4.tcp_keepalive_time = 600
  14. net.ipv4.tcp_keepalive_probes = 3
  15. net.ipv4.tcp_keepalive_intvl =15
  16. net.ipv4.tcp_max_tw_buckets = 36000
  17. net.ipv4.tcp_tw_reuse = 1
  18. net.ipv4.tcp_max_orphans = 327680
  19. net.ipv4.tcp_orphan_retries = 3
  20. net.ipv4.tcp_syncookies = 1
  21. net.ipv4.tcp_max_syn_backlog = 16384
  22. net.ipv4.ip_conntrack_max = 65536
  23. net.ipv4.tcp_max_syn_backlog = 16384
  24. net.ipv4.tcp_timestamps = 0
  25. net.core.somaxconn = 16384
  26. EOF
  27. sysctl --system
  1. 1.1.3  基本组件安装
  2. 本节主要安装的是集群中用到的各种组件，比如Docker-ce、Kubernetes各组件等。
  3. 查看可用docker-ce版本：
  4. yum list docker-ce.x86_64 --showduplicates | sort -r
  5. [root@k8s-master01 k8s-ha-install]# wget https://download.docker.com/linux/centos/7/x86_64/edge/Packages/containerd.io-1.2.13-3.2.el7.x86_64.rpm 
  7. 安装 docker-ce 19.03  版本：
  9. yum install -y docker-ce-cli-19.03.8-3.el7.x86_64 docker-ce-19.03.8-3.el7.x86_64
  1. 温馨提示：
  2. 由于新版kubelet建议使用systemd，所以可以把docker的CgroupDriver改成systemd
  3. cat > /etc/docker/daemon.json <<EOF
  4. {
  5.   "exec-opts": ["native.cgroupdriver=systemd"]
  6. }
  7. EOF

image_1f1mj9sue1if77h9op982o48n16.png-195.2kB

  1. 启动docker 
  3. service docker start
  4. chkconfig docker on

image_1f1mjbmf21bbjpftc3e1vqit7r1j.png-141.1kB

  1. 安装k8s组件：
  2. yum list kubeadm.x86_64 --showduplicates | sort -r
  3.     所有节点安装最新版本kubeadm：
  4. yum install kubeadm -y
  5. 所有节点安装指定版本k8s组件：
  6. yum install -y kubeadm-1.20.5-0.x86_64 kubectl-1.20.5-0.x86_64 kubelet-1.20.5-0.x86_64
  7. 所有节点设置开机自启动Docker：
  8. systemctl daemon-reload && systemctl enable --now docker
  9. 默认配置的pause镜像使用gcr.io仓库，国内可能无法访问，所以这里配置Kubelet使用阿里云的pause镜像：
  11. DOCKER_CGROUPS=$(docker info | grep 'Cgroup' | cut -d' ' -f4)
  12. cat >/etc/sysconfig/kubelet<<EOF
  13. KUBELET_EXTRA_ARGS="--cgroup-driver=$DOCKER_CGROUPS --pod-infra-container-image=registry.cn-hangzhou.aliyuncs.com/google_containers/pause-amd64:3.1"
  14. EOF

image_1f1mjfeet1qk215s96eg88h1oj820.png-257.3kB

image_1f1mjhmfttl5m4k1a1s1ih5rkr2d.png-88.8kB

--

  1. 设置Kubelet开机自启动：
  2. systemctl daemon-reload
  3. systemctl enable --now kubelet
  1. 1.1.4  高可用组件安装
  2. 所有Master节点通过yum安装HAProxy和KeepAlived：
  3. yum install keepalived haproxy -y
  1. 所有Master节点配置HAProxy（详细配置参考HAProxy文档，所有Master节点的HAProxy配置相同）：
  2. [root@k8s-master01 etc]# mkdir /etc/haproxy
  3. [root@k8s-master01 etc]# vim /etc/haproxy/haproxy.cfg 
  4. global
  5.   maxconn  2000
  6.   ulimit-n  16384
  7.   log  127.0.0.1 local0 err
  8.   stats timeout 30s
  10. defaults
  11.   log global
  12.   mode  http
  13.   option  httplog
  14.   timeout connect 5000
  15.   timeout client  50000
  16.   timeout server  50000
  17.   timeout http-request 15s
  18.   timeout http-keep-alive 15s
  20. frontend monitor-in
  21.   bind *:33305
  22.   mode http
  23.   option httplog
  24.   monitor-uri /monitor
  26. listen stats
  27.   bind    *:8006
  28.   mode    http
  29.   stats   enable
  30.   stats   hide-version
  31.   stats   uri       /stats
  32.   stats   refresh   30s
  33.   stats   realm     Haproxy\ Statistics
  34.   stats   auth      admin:admin
  36. frontend k8s-master
  37.   bind 0.0.0.0:16443
  38.   bind 127.0.0.1:16443
  39.   mode tcp
  40.   option tcplog
  41.   tcp-request inspect-delay 5s
  42.   default_backend k8s-master
  44. backend k8s-master
  45.   mode tcp
  46.   option tcplog
  47.   option tcp-check
  48.   balance roundrobin
  49.   default-server inter 10s downinter 5s rise 2 fall 2 slowstart 60s maxconn 250 maxqueue 256 weight 100
  50.   server node01.flyfish.cn      192.168.100.11:6443  check
  51.   server node02.flyfish.cn      192.168.100.12:6443  check
  52.   server node03.flyfish.cn      192.168.100.13:6443  check
  54.   ----
  56.   三台机器的配置是一样的：
  58.   scp  haproxy.cfg root@node02.flyfish.cn:/etc/haproxy/
  59.   scp  haproxy.cfg root@node03.flyfish.cn:/etc/haproxy/

image_1f1mk5gv41epj1kar2kq1ou41fsf2q.png-84.6kB

  1. Master01节点的配置：
  2. [root@k8s-master01 etc]# mkdir /etc/keepalived
  4. [root@k8s-master01 ~]# vim /etc/keepalived/keepalived.conf 
  5. ! Configuration File for keepalived
  6. global_defs {
  7.     router_id LVS_DEVEL
  8. }
  9. vrrp_script chk_apiserver {
  10.     script "/etc/keepalived/check_apiserver.sh"
  11.     interval 2
  12.     weight -5
  13.     fall 3  
  14.     rise 2
  15. }
  16. vrrp_instance VI_1 {
  17.     state MASTER
  18.     interface ens33
  19.     mcast_src_ip 192.168.100.11
  20.     virtual_router_id 51
  21.     priority 100
  22.     advert_int 2
  23.     authentication {
  24.         auth_type PASS
  25.         auth_pass K8SHA_KA_AUTH
  26.     }
  27.     virtual_ipaddress {
  28.         192.168.100.200
  29.     }
  30.     track_script {
  31.        chk_apiserver
  32.     }
  33. }
  1. Master02节点的配置：
  2. ! Configuration File for keepalived
  3. global_defs {
  4.     router_id LVS_DEVEL
  5. }
  6. vrrp_script chk_apiserver {
  7.     script "/etc/keepalived/check_apiserver.sh"
  8.     interval 2
  9.     weight -5
  10.     fall 3  
  11.     rise 2
  12. }
  13. vrrp_instance VI_1 {
  14.     state BACKUP
  15.     interface ens33
  16.     mcast_src_ip 192.168.100.12
  17.     virtual_router_id 51
  18.     priority 101
  19.     advert_int 2
  20.     authentication {
  21.         auth_type PASS
  22.         auth_pass K8SHA_KA_AUTH
  23.     }
  24.     virtual_ipaddress {
  25.         192.168.100.200
  26.     }
  27.     track_script {
  28.        chk_apiserver
  29.     }
  30. }
  1. Master03节点的配置：
  2. ! Configuration File for keepalived
  3. global_defs {
  4.     router_id LVS_DEVEL
  5. }
  6. vrrp_script chk_apiserver {
  7.     script "/etc/keepalived/check_apiserver.sh"
  8.     interval 2
  9.     weight -5
  10.     fall 3  
  11.     rise 2
  12. }
  13. vrrp_instance VI_1 {
  14.     state BACKUP
  15.     interface ens33
  16.     mcast_src_ip 192.168.100.13
  17.     virtual_router_id 51
  18.     priority 102
  19.     advert_int 2
  20.     authentication {
  21.         auth_type PASS
  22.         auth_pass K8SHA_KA_AUTH
  23.     }
  24.     virtual_ipaddress {
  25.         192.168.100.200
  26.     }
  27.     track_script {
  28.        chk_apiserver
  29.     }
  30. }
  32. 注意上述的健康检查是关闭的，集群建立完成后再开启：
  33.     track_script {
  34.        chk_apiserver
  35.     }
  1. 配置KeepAlived健康检查文件：
  2. [root@k8s-master01 keepalived]# cat /etc/keepalived/check_apiserver.sh 
  3. #!/bin/bash
  5. err=0
  6. for k in $(seq 1 5)
  7. do
  8.     check_code=$(pgrep kube-apiserver)
  9.     if [[ $check_code == "" ]]; then
  10.         err=$(expr $err + 1)
  11.         sleep 5
  12.         continue
  13.     else
  14.         err=0
  15.         break
  16.     fi
  17. done
  19. if [[ $err != "0" ]]; then
  20.     echo "systemctl stop keepalived"
  21.     /usr/bin/systemctl stop keepalived
  22.     exit 1
  23. else
  24.     exit 0
  25. fi
  1. 启动haproxy和keepalived (所有master 启动)
  2. [root@k8s-master01 keepalived]# systemctl enable --now haproxy
  3. [root@k8s-master01 keepalived]# systemctl enable --now keepalived

image_1f1mldn831cbj1g601idu7opqhv37.png-175.2kB

  1. 集群初始化：
  3. https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/
  1. 各Master节点的kubeadm-config.yaml配置文件如下：
  2. Master01：
  3. daocloud.io/daocloud
  4. -----
  5. apiVersion: kubeadm.k8s.io/v1beta2
  6. bootstrapTokens:
  7. - groups:
  8.   - system:bootstrappers:kubeadm:default-node-token
  9.   token: 7t2weq.bjbawausm0jaxury
  10.   ttl: 24h0m0s
  11.   usages:
  12.   - signing
  13.   - authentication
  14. kind: InitConfiguration
  15. localAPIEndpoint:
  16.   advertiseAddress: 192.168.100.11
  17.   bindPort: 6443
  18. nodeRegistration:
  19.   criSocket: /var/run/dockershim.sock
  20.   name: node01.flyfish.cn
  21.   taints:
  22.   - effect: NoSchedule
  23.     key: node-role.kubernetes.io/master
  24. ---
  25. apiServer:
  26.   certSANs:
  27.   - 192.168.100.200
  28.   timeoutForControlPlane: 4m0s
  29. apiVersion: kubeadm.k8s.io/v1beta2
  30. certificatesDir: /etc/kubernetes/pki
  31. clusterName: kubernetes
  32. controlPlaneEndpoint: 192.168.100.200:16443
  33. controllerManager: {}
  34. dns:
  35.   type: CoreDNS
  36. etcd:
  37.   local:
  38.     dataDir: /var/lib/etcd
  39. imageRepository: registry.cn-hangzhou.aliyuncs.com/google_containers
  40. kind: ClusterConfiguration
  41. kubernetesVersion: v1.20.5
  42. networking:
  43.   dnsDomain: cluster.local
  44.   podSubnet: 172.168.100.0/16
  45.   serviceSubnet: 10.96.0.0/12
  46. scheduler: {}
  47. ----
  1. 更新kubeadm文件
  2. kubeadm config migrate --old-config kubeadm-config.yaml --new-config new.yaml
  3. 所有Master节点提前下载镜像，可以节省初始化时间：(master 节点)
  4. kubeadm config images pull --config /root/new.yaml
  5. 所有节点设置开机自启动kubelet
  6. systemctl enable --now kubelet

image_1f1mn8j171c4e1v8l1t051h1j1ibqm.png-35.8kB

image_1f1mn7ikttv911qe1nneifbgif9.png-108.7kB

  1. Master01节点初始化，初始化以后会在/etc/kubernetes目录下生成对应的证书和配置文件，之后其他Master节点加入Master01即可：
  2. kubeadm init --config /root/kubeadm-config.yaml  --upload-certs
  4. 不用配置文件初始化：kubeadm init --control-plane-endpoint "LOAD_BALANCER_DNS:LOAD_BALANCER_PORT" --upload-certs
  1. 初始化失败报错
  2.     error execution phase upload-config/kubelet: Error writing Crisocket information for the control-plane node: timed out waiting for the condition
  3. To see the stack trace of this error execute with --v=5 or higher

image_1f1mq60mv7411bmk7nf1l6b1udr2j.png-134.9kB

  1. 解决方法：
  2.        所有主机停掉kubelet 
  3.          service kubelet stop 
  5.          执行命令：
  7.   swapoff -a && kubeadm reset  && systemctl daemon-reload && systemctl restart kubelet  && iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X
  8.    ipvsadm --clear

image_1f1mqabg41t591dva1329rvn1j5q33.png-251kB

  1. 再次初始化：
  3.     kubeadm init --config /root/new.yaml --upload-certs

image_1f1mqd2116eu1qa1jbp1819oqr3g.png-371.9kB

image_1f1mqdlqh18smvcpg9dsf3mf63t.png-322.9kB

  1. Your Kubernetes control-plane has initialized successfully!
  3. To start using your cluster, you need to run the following as a regular user:
  5.   mkdir -p $HOME/.kube
  6.   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  7.   sudo chown $(id -u):$(id -g) $HOME/.kube/config
  9. Alternatively, if you are the root user, you can run:
  11.   export KUBECONFIG=/etc/kubernetes/admin.conf
  13. You should now deploy a pod network to the cluster.
  14. Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  15.   https://kubernetes.io/docs/concepts/cluster-administration/addons/
  17. You can now join any number of the control-plane node running the following command on each as root:
  19.   kubeadm join 192.168.100.200:16443 --token 7t2weq.bjbawausm0jaxury \
  20.     --discovery-token-ca-cert-hash sha256:7263545b1a028e6217ff4e55712bf24422e6d9aeba54e76daabfc8a824ffcd99 \
  21.     --control-plane --certificate-key c0b3b67c42f4fe9ae2832d86f80df35ee2e7b32f945906fabe60e4ae1f4ba18f
  23. Please note that the certificate-key gives access to cluster sensitive data, keep it secret!
  24. As a safeguard, uploaded-certs will be deleted in two hours; If necessary, you can use
  25. "kubeadm init phase upload-certs --upload-certs" to reload certs afterward.
  27. Then you can join any number of worker nodes by running the following on each as root:
  29. kubeadm join 192.168.100.200:16443 --token 7t2weq.bjbawausm0jaxury \
  30.     --discovery-token-ca-cert-hash sha256:7263545b1a028e6217ff4e55712bf24422e6d9aeba54e76daabfc8a824ffcd99
  1.   mkdir -p $HOME/.kube
  2.   sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  3.   sudo chown $(id -u):$(id -g) $HOME/.kube/config
  1. 所有Master节点配置环境变量，用于访问Kubernetes集群：
  2. cat <<EOF >> /root/.bashrc
  3. export KUBECONFIG=/etc/kubernetes/admin.conf
  4. EOF
  5. source /root/.bashrc
  6. 查看节点状态：
  7. [root@k8s-master01 ~]# kubectl get nodes
  8. NAME           STATUS     ROLES     AGE       VERSION
  9. k8s-master01   NotReady   master    14m       v1.12.3
  10. 采用初始化安装方式，所有的系统组件均以容器的方式运行并且在kube-system命名空间内，此时可以查看Pod状态：
  1. 其他master 加入集群：
  2.      kubeadm join 192.168.100.200:16443 --token 7t2weq.bjbawausm0jaxury \
  3.     --discovery-token-ca-cert-hash sha256:7263545b1a028e6217ff4e55712bf24422e6d9aeba54e76daabfc8a824ffcd99 \
  4.     --control-plane --certificate-key c0b3b67c42f4fe9ae2832d86f80df35ee2e7b32f945906fabe60e4ae1f4ba18f

image_1f1ms7272ufi33bcm1bit1b7e4q.png-274.5kB

  1. mkdir -p $HOME/.kube
  2. sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  3. sudo chown $(id -u):$(id -g) $HOME/.kube/config
  1. 加master节点报错
  2.    [ERROR DirAvailable--var-lib-etcd]: /var/lib/etcd is not empty
  3. [preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
  4. To see the stack trace of this error execute with --v=5 or higher

image_1f1msjrle1p9817bd1u2etq0i1u57.png-115kB

  1. rm -rf /var/lib/etcd
  1. 在初始化：
  2.     kubeadm join 192.168.100.200:16443 --token 7t2weq.bjbawausm0jaxury \
  3.     --discovery-token-ca-cert-hash sha256:7263545b1a028e6217ff4e55712bf24422e6d9aeba54e76daabfc8a824ffcd99 \
  4.     --control-plane --certificate-key c0b3b67c42f4fe9ae2832d86f80df35ee2e7b32f945906fabe60e4ae1f4ba18f

image_1f1msni0f1vnbj3qvkeige1i6k5k.png-371.2kB

  1. mkdir -p $HOME/.kube
  2. sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  3. sudo chown $(id -u):$(id -g) $HOME/.kube/config

image_1f1msorv8stapetkm8nfm1ejp61.png-119.5kB

  1. kubectl get node

image_1f1msplr71chm105p5k5bkfq0i6e.png-72.6kB

  1. node 节点加入：
  2.        kubeadm join 192.168.100.200:16443 --token 7t2weq.bjbawausm0jaxury \
  3.     --discovery-token-ca-cert-hash sha256:7263545b1a028e6217ff4e55712bf24422e6d9aeba54e76daabfc8a824ffcd99

image_1f1msu790akc19nqrce57k1lnr6r.png-129.4kB

  1. Token过期后生成新的token：(集群扩容与缩容的问题)
  2. kubeadm token create --print-join-command 
  3. -----
  4. kubeadm join 192.168.100.200:16443 --token p6rvkq.0joqbi5bxnd12n20     --discovery-token-ca-cert-hash sha256:7263545b1a028e6217ff4e55712bf24422e6d9aeba54e76daabfc8a824ffcd99
  6. -----
  7. Master需要生成--certificate-key
  8. kubeadm init phase upload-certs  --upload-certs
  9. -----
  10. [upload-certs] Storing the certificates in Secret "kubeadm-certs" in the "kube-system" Namespace
  11. [upload-certs] Using certificate key:
  12. c97af03609a9d3c099dd07615ce170a8f0c8db368f36e10a79576f9efcc0857e
  13. ----
  15. 初始化其他master加入集群
  16.    kubeadm join 192.168.100.200:16443 --token p6rvkq.0joqbi5bxnd12n20     --discovery-token-ca-cert-hash sha256:7263545b1a028e6217ff4e55712bf24422e6d9aeba54e76daabfc8a824ffcd99 --control-plane --certificate-key c97af03609a9d3c099dd07615ce170a8f0c8db368f36e10a79576f9efcc0857e

image_1f1mu5m9711nt11o4g6r1p9u1j7878.png-103.8kB

  1. 查看token的过期时间
  2.     kubectl get secret -n kube-system -o wide 
  1. kubectl get secret -n kube-system bootstrap-token-q8soir -o yaml
  2. -----
  3. apiVersion: v1
  4. data:
  5.   description: UHJveHkgZm9yIG1hbmFnaW5nIFRUTCBmb3IgdGhlIGt1YmVhZG0tY2VydHMgc2VjcmV0
  6.   expiration: MjAyMS0wMy0yNlQxODoyMzowNSswODowMA==
  7.   token-id: cThzb2ly
  8.   token-secret: cnZidjZwdGIxajIxbWFsZw==
  9. kind: Secret
  10. metadata:
  11.   creationTimestamp: "2021-03-26T08:23:05Z"
  12.   managedFields:
  13.   - apiVersion: v1
  14.     fieldsType: FieldsV1
  15.     fieldsV1:
  16.       f:data:
  17.         .: {}
  18.         f:description: {}
  19.         f:expiration: {}
  20.         f:token-id: {}
  21.         f:token-secret: {}
  22.       f:type: {}
  23.     manager: kubeadm
  24.     operation: Update
  25.     time: "2021-03-26T08:23:05Z"
  26.   name: bootstrap-token-q8soir
  27.   namespace: kube-system
  28.   resourceVersion: "305"
  29.   uid: eba712c8-386e-4824-802e-7a2f5cd05f0c
  30. type: bootstrap.kubernetes.io/token
  31. ----

image_1f1mug5h37741ib913ruksd170g7l.png-158.2kB

  1. echo MjAyMS0wMy0yNlQxODoyMzowNSswODowMA== |base64 -d

image_1f1muher87hf4ebbs718ak17cp82.png-196kB

  1. kubectl get node

image_1f1muodoa1anri996om1eslnro8f.png-73.9kB

  1. 修改calico-etcd.yaml的以下位置
  2.      cd /root/k8s-ha-install && git checkout manual-installation-v1.20.x && cd calico/
  3. -----
  4. sed -i 's#etcd_endpoints: "http://<ETCD_IP>:<ETCD_PORT>"#etcd_endpoints: "https://192.168.100.11:2379,https://192.168.100.12:2379,https://192.168.100.13:2379"#g' calico-etcd.yaml
  6. ETCD_CA=`cat /etc/kubernetes/pki/etcd/ca.crt | base64 | tr -d '\n'`
  8. ETCD_CERT=`cat /etc/kubernetes/pki/etcd/server.crt | base64 | tr -d '\n'`
  10. ETCD_KEY=`cat /etc/kubernetes/pki/etcd/server.key | base64 | tr -d '\n'`
  12. sed -i "s@# etcd-key: null@etcd-key: ${ETCD_KEY}@g; s@# etcd-cert: null@etcd-cert: ${ETCD_CERT}@g; s@# etcd-ca: null@etcd-ca: ${ETCD_CA}@g" calico-etcd.yaml
  14. sed -i 's#etcd_ca: ""#etcd_ca: "/calico-secrets/etcd-ca"#g; s#etcd_cert: ""#etcd_cert: "/calico-secrets/etcd-cert"#g; s#etcd_key: "" #etcd_key: "/calico-secrets/etcd-key" #g' calico-etcd.yaml
  16. POD_SUBNET=`cat /etc/kubernetes/manifests/kube-controller-manager.yaml | grep cluster-cidr= | awk -F= '{print $NF}'`
  18. sed -i 's@# - name: CALICO_IPV4POOL_CIDR@- name: CALICO_IPV4POOL_CIDR@g; s@#   value: "192.168.0.0/16"@  value: '"${POD_SUBNET}"'@g' calico-etcd.yaml
  20. -----
  22. kubectl apply -f  calico-etcd.yaml
  24. kubectl get node -n kue-system 
  26. kubectl get node

image_1f1n1am3q1evr1lnh1eti1l0c166a8s.png-201.3kB

image_1f1n1bivn15lkptmvda15sk1f1999.png-65.3kB

  1. 配置metric server 
  2. cd /root/k8s-ha-install/metrics-server-0.4.x-kubeadm
  3. vim comp.ymal
  4. ----
  5. apiVersion: v1
  6. kind: ServiceAccount
  7. metadata:
  8.   labels:
  9.     k8s-app: metrics-server
  10.   name: metrics-server
  11.   namespace: kube-system
  12. ---
  13. apiVersion: rbac.authorization.k8s.io/v1
  14. kind: ClusterRole
  15. metadata:
  16.   labels:
  17.     k8s-app: metrics-server
  18.     rbac.authorization.k8s.io/aggregate-to-admin: "true"
  19.     rbac.authorization.k8s.io/aggregate-to-edit: "true"
  20.     rbac.authorization.k8s.io/aggregate-to-view: "true"
  21.   name: system:aggregated-metrics-reader
  22. rules:
  23.   - apiGroups:
  24.       - metrics.k8s.io
  25.     resources:
  26.       - pods
  27.       - nodes
  28.     verbs:
  29.       - get
  30.       - list
  31.       - watch
  32. ---
  33. apiVersion: rbac.authorization.k8s.io/v1
  34. kind: ClusterRole
  35. metadata:
  36.   labels:
  37.     k8s-app: metrics-server
  38.   name: system:metrics-server
  39. rules:
  40.   - apiGroups:
  41.       - ""
  42.     resources:
  43.       - pods
  44.       - nodes
  45.       - nodes/stats
  46.       - namespaces
  47.       - configmaps
  48.     verbs:
  49.       - get
  50.       - list
  51.       - watch
  52. ---
  53. apiVersion: rbac.authorization.k8s.io/v1
  54. kind: RoleBinding
  55. metadata:
  56.   labels:
  57.     k8s-app: metrics-server
  58.   name: metrics-server-auth-reader
  59.   namespace: kube-system
  60. roleRef:
  61.   apiGroup: rbac.authorization.k8s.io
  62.   kind: Role
  63.   name: extension-apiserver-authentication-reader
  64. subjects:
  65.   - kind: ServiceAccount
  66.     name: metrics-server
  67.     namespace: kube-system
  68. ---
  69. apiVersion: rbac.authorization.k8s.io/v1
  70. kind: ClusterRoleBinding
  71. metadata:
  72.   labels:
  73.     k8s-app: metrics-server
  74.   name: metrics-server:system:auth-delegator
  75. roleRef:
  76.   apiGroup: rbac.authorization.k8s.io
  77.   kind: ClusterRole
  78.   name: system:auth-delegator
  79. subjects:
  80.   - kind: ServiceAccount
  81.     name: metrics-server
  82.     namespace: kube-system
  83. ---
  84. apiVersion: rbac.authorization.k8s.io/v1
  85. kind: ClusterRoleBinding
  86. metadata:
  87.   labels:
  88.     k8s-app: metrics-server
  89.   name: system:metrics-server
  90. roleRef:
  91.   apiGroup: rbac.authorization.k8s.io
  92.   kind: ClusterRole
  93.   name: system:metrics-server
  94. subjects:
  95.   - kind: ServiceAccount
  96.     name: metrics-server
  97.     namespace: kube-system
  98. ---
  99. apiVersion: v1
  100. kind: Service
  101. metadata:
  102.   labels:
  103.     k8s-app: metrics-server
  104.   name: metrics-server
  105.   namespace: kube-system
  106. spec:
  107.   ports:
  108.     - name: https
  109.       port: 443
  110.       protocol: TCP
  111.       targetPort: https
  112.   selector:
  113.     k8s-app: metrics-server
  114. ---
  115. apiVersion: apps/v1
  116. kind: Deployment
  117. metadata:
  118.   labels:
  119.     k8s-app: metrics-server
  120.   name: metrics-server
  121.   namespace: kube-system
  122. spec:
  123.   selector:
  124.     matchLabels:
  125.       k8s-app: metrics-server
  126.   strategy:
  127.     rollingUpdate:
  128.       maxUnavailable: 0
  129.   template:
  130.     metadata:
  131.       labels:
  132.         k8s-app: metrics-server
  133.     spec:
  134.       containers:
  135.         - args:
  136.             - --cert-dir=/tmp
  137.             - --secure-port=4443
  138.             - --metric-resolution=30s
  139.             - --kubelet-insecure-tls
  140.             - --kubelet-preferred-address-types=InternalIP,ExternalIP,Hostname
  141.           #  - --requestheader-client-ca-file=/etc/kubernetes/pki/front-proxy-ca.crt # change to front-proxy-ca.crt for kubeadm
  142.             - --requestheader-username-headers=X-Remote-User
  143.             - --requestheader-group-headers=X-Remote-Group
  144.             - --requestheader-extra-headers-prefix=X-Remote-Extra-
  145.           image: registry.cn-beijing.aliyuncs.com/dotbalo/metrics-server:v0.4.1
  146.           imagePullPolicy: IfNotPresent
  147.           livenessProbe:
  148.             failureThreshold: 3
  149.             httpGet:
  150.               path: /livez
  151.               port: https
  152.               scheme: HTTPS
  153.             periodSeconds: 10
  154.           name: metrics-server
  155.           ports:
  156.             - containerPort: 4443
  157.               name: https
  158.               protocol: TCP
  159.           readinessProbe:
  160.             failureThreshold: 3
  161.             httpGet:
  162.               path: /readyz
  163.               port: https
  164.               scheme: HTTPS
  165.             periodSeconds: 10
  166.           securityContext:
  167.             readOnlyRootFilesystem: true
  168.             runAsNonRoot: true
  169.             runAsUser: 1000
  170.           volumeMounts:
  171.             - mountPath: /tmp
  172.               name: tmp-dir
  173.             - name: ca-ssl
  174.               mountPath: /etc/kubernetes/pki
  175.       nodeSelector:
  176.         kubernetes.io/os: linux
  177.       priorityClassName: system-cluster-critical
  178.       serviceAccountName: metrics-server
  179.       volumes:
  180.         - emptyDir: {}
  181.           name: tmp-dir
  182.         - name: ca-ssl
  183.           hostPath:
  184.             path: /etc/kubernetes/pki
  185. ---
  186. apiVersion: apiregistration.k8s.io/v1
  187. kind: APIService
  188. metadata:
  189.   labels:
  190.     k8s-app: metrics-server
  191.   name: v1beta1.metrics.k8s.io
  192. spec:
  193.   group: metrics.k8s.io
  194.   groupPriorityMinimum: 100
  195.   insecureSkipTLSVerify: true
  196.   service:
  197.     name: metrics-server
  198.     namespace: kube-system
  199.   version: v1beta1
  200.   versionPriority: 100
  202. ----
  203. kubectl apply -f comp.yaml

image_1f9m4197m14vmmup1tfiede183d9.png-140.2kB

image_1f9m42fcr15681hpi1cmn7st1d59m.png-253.4kB

  1. 安装dashbaord 
  3. cd /root/k8s-ha-install/dashboard
  4. kubectl apply -f dashboard-user.yaml
  5. kubectl apply -f dashboard.yaml
  7. kubectl get pod -n kubernetes-dashboard

image_1f9m4bogl9g21v231tc16aev0e13.png-133.5kB

  1. kubectl edit svc kubernetes-dashboard -n kubernetes-dashboard
  2. 改一下svc 的 类型：
  3.    type: Cluster-IP
  4.    改为： type: NodePort

image_1f9m4p2ed167q8r51vbr1n83lbj1g.png-36.5kB

  1. kubectl get svc -n kubernetes-dashboard -o wide

image_1f9m4qfrlvign6k1venf2p12gj2d.png-105.9kB

  1. kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

image_1f9m4tjr7jnbd7p1rlk1lv911n5a.png-235.4kB

image_1f9m4vn2219281hupojg8or14le5n.png-271kB

image_1f9m52b078d91ei3p0cu351eqp8k.png-450kB

image_1f9m64ouv1slimm81t5k1f5u15r391.png-379.7kB

  1. 集群测试：
  2.    kubectl get svc -n kube-system 
  4.    telnet 10.96.0.1 443 
  5.    telnet 10.96.0.10 53

image_1f9mf904t1grm1akf1nt51b1v1g879.png-103.6kB

image_1f9mflqut1moj1dqh9ojbko1pju26.png-180.5kB

  1. 部署kuborad 
  2. 在node 节点上面 下载镜像：
  3.   docker pull  eipwork/kuboard:latest
  5.   kubectl apply -f https://kuboard.cn/install-script/kuboard.yaml
  7.   kubectl get pods -l k8s.kuboard.cn/name=kuboard -n kube-system
  9. kubectl get svc,pod -n kube-system -o wide |grep kuboard

image_1f9n6n32h1iko1st01r792si9fa9.png-80.8kB

image_1f9n6tps337kp864141oq21v9p13.png-93.3kB

  1. 获取Token
  3. # 如果您参考 www.kuboard.cn 提供的文档安装 Kuberenetes，可在第一个 Master 节点上执行此命令
  5. echo $(kubectl -n kube-system get secret $(kubectl -n kube-system get secret | grep kuboard-user | awk '{print $1}') -o go-template='{{.data.token}}' | base64 -d)

image_1f9n72ituq841b7nfrkare1jlp1g.png-188.7kB

image_1f9n73hqf1a24qkjsf37ms12et1t.png-406.6kB

image_1f9n74aga151hkhfknnhlts3n2a.png-563.5kB

  1. 卸载：kuborad-v2
  2.   kubectl delete -f https://kuboard.cn/install-script/kuboard.yaml
  1. 安装kuboard-v3
  2. 在node 节点上面下载镜像：
  3. docker pull eipwork/kuboard:v3
  4. docker pull eipwork/etcd-host:3.4.16-1
  5. mkdir /data
  6. chmod 777 -R /data
  8. 配置镜像下载策略
  9. wget https://addons.kuboard.cn/kuboard/kuboard-v3.yaml
  10. vim kuboard-v3.ymal
  11. ---
  12. imagePullPolicy: IfNotPresent  (共有两处)
  13. ---
  15. kubectl apply -f kuboard-v3.yaml

image_1f9n87u3i12901tq0ad71ku71er52n.png-95.4kB

image_1f9n89h5e1sbs4h41pf7fldusl34.png-61.9kB

image_1f9n8b1nk1lv114pg1o3h1hbpt1b3h.png-893.7kB

  1. 访问 Kuboard
  2. 在浏览器中打开链接 http://your-node-ip-address:30080
  4. 输入初始用户名和密码，并登录
  6. 用户名： admin
  7. 密码： Kuboard123

image_1f9n8hpeu1b3m1k7v1vbhj3r1h4j4r.png-587.9kB

image_1f9n8im0u151hdaaikg1fnm1b5458.png-318.2kB

image_1f9n8jepj1mf216m79rj13vv193b65.png-493.2kB

image_1f9n8k56p1coc17mt1ohn1bu3n3j6i.png-669.4kB

image_1f9n8gaiu1587td1qstukp2nt3u.png-518.7kB

添加新批注
在作者公开此批注前，只有你和作者可见。
回复批注