k8s1.20.x 二进制高可用部署

kubernetes架构系列

一：系统环境初始化

1.1 系统环境

系统:
centos7.9x64-1
主机名：
cat /etc/hosts 
----
192.168.100.11  k8s-master01
192.168.100.12  k8s-master02
192.168.100.13  k8s-master03
192.168.100.100 k8s-master-lb
192.168.100.14  k8s-node01
192.168.100.15  k8s-node02
192.168.100.16  k8s-node03
----
k8s 1.20.x的 高可用

1.2 系统环境初始化

所有节点关闭firewalld 、dnsmasq、selinux(NetworkManager):
systemctl disable --now firewalld 
systemctl disable --now dnsmasq
setenforce 0
所有节点关闭swap分区:
swapoff -a && sysctl -w vm.swappiness=0

时间服务器：
vim /etc/chrony.conf
----
service ntp1.aliyun.com iburst
----
service chronyd stop 
service chronyd start
chronyc sources -v 

所有节点安装基本工具
yum install wget jq psmisc vim net-tools yum-utils device-mapper-persistent-data lvm2 git -y

下载所需要文件：
  git clone https://github.com/dotbalo/k8s-ha-install.git
  git branch -a 
  git checkout manual-installation-v1.20.x

1.3 安装 源所需准备

centos7的源安装： 【全部节点】
curl -o /etc/yum.repos.d/CentOS-Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
yum install -y yum-utils device-mapper-persistent-data lvm2
yum-config-manager --add-repo https://mirrors.aliyun.com/docker-ce/linux/centos/docker-ce.repo

CentOS 8 安装源如下：
curl -o /etc/yum.repos.d/CentOS-Base.repo http://mirrors.aliyun.com/repo/Centos-8.repo
yum makecache
yum install -y yum-utils device-mapper-persistent-data lvm2

所有节点安装ipvs
yum install ipvsadm ipset sysstat conntrack libseccomp -y

vim /etc/modules-load.d/ipvs.conf
ip_vs
ip_vs_lc
ip_vs_wlc
ip_vs_rr
ip_vs_wrr
ip_vs_lblc
ip_vs_lblcr
ip_vs_dh
ip_vs_sh
ip_vs_fo
ip_vs_nq
ip_vs_sed
ip_vs_ftp
ip_vs_sh
nf_conntrack_ipv4
ip_tables
ip_set
xt_set
ipt_set
ipt_rpfilter
ipt_REJECT
ipip
sysctl -p 
systemctl enable --now systemd-modules-load.service
reboot 重启所有机器

1.5 系统句柄准备

cat <<EOF > /etc/sysctl.d/k8s.conf
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
fs.may_detach_mounts = 1
vm.overcommit_memory=1
vm.panic_on_oom=0
fs.inotify.max_user_watches=89100
fs.file-max=52706963
fs.nr_open=52706963
net.netfilter.nf_conntrack_max=2310720
net.ipv4.tcp_keepalive_time = 600
net.ipv4.tcp_keepalive_probes = 3
net.ipv4.tcp_keepalive_intvl =15
net.ipv4.tcp_max_tw_buckets = 36000
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_max_orphans = 327680
net.ipv4.tcp_orphan_retries = 3
net.ipv4.tcp_syncookies = 1
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.ip_conntrack_max = 65536
net.ipv4.tcp_max_syn_backlog = 16384
net.ipv4.tcp_timestamps = 0
net.core.somaxconn = 16384
EOF
sysctl --system

1.5 系统内核升级准备

如需内核升级，可以按需操作：
CentOS7 需要升级内核至4.18+
https://www.kernel.org/ 和 https://elrepo.org/linux/kernel/el7/x86_64/
CentOS 7 dnf可能无法安装内核
dnf --disablerepo=\* --enablerepo=elrepo -y install kernel-ml kernel-ml-devel
grubby --default-kernel
使用如下方式安装最新版内核
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
rpm -Uvh http://www.elrepo.org/elrepo-release-7.0-2.el7.elrepo.noarch.rpm
查看最新版内核yum --disablerepo="*" --enablerepo="elrepo-kernel" list available
[root@k8s-node01 ~]# yum --disablerepo="*" --enablerepo="elrepo-kernel" list available
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * elrepo-kernel: mirrors.neusoft.edu.cn
elrepo-kernel                                                                                  | 2.9 kB  00:00:00     
elrepo-kernel/primary_db                                                                       | 1.9 MB  00:00:00     
Available Packages
elrepo-release.noarch                                      7.0-5.el7.elrepo                              elrepo-kernel
kernel-lt.x86_64                                           4.4.229-1.el7.elrepo                          elrepo-kernel
kernel-lt-devel.x86_64                                     4.4.229-1.el7.elrepo                          elrepo-kernel
kernel-lt-doc.noarch                                       4.4.229-1.el7.elrepo                          elrepo-kernel
kernel-lt-headers.x86_64                                   4.4.229-1.el7.elrepo                          elrepo-kernel
kernel-lt-tools.x86_64                                     4.4.229-1.el7.elrepo                          elrepo-kernel
kernel-lt-tools-libs.x86_64                                4.4.229-1.el7.elrepo                          elrepo-kernel
kernel-lt-tools-libs-devel.x86_64                          4.4.229-1.el7.elrepo                          elrepo-kernel
kernel-ml.x86_64                                           5.7.7-1.el7.elrepo                            elrepo-kernel
kernel-ml-devel.x86_64                                     5.7.7-1.el7.elrepo                            elrepo-kernel
kernel-ml-doc.noarch                                       5.7.7-1.el7.elrepo                            elrepo-kernel
kernel-ml-headers.x86_64                                   5.7.7-1.el7.elrepo                            elrepo-kernel
kernel-ml-tools.x86_64                                     5.7.7-1.el7.elrepo                            elrepo-kernel
kernel-ml-tools-libs.x86_64                                5.7.7-1.el7.elrepo                            elrepo-kernel
kernel-ml-tools-libs-devel.x86_64                          5.7.7-1.el7.elrepo                            elrepo-kernel
perf.x86_64                                                5.7.7-1.el7.elrepo                            elrepo-kernel
python-perf.x86_64                                         5.7.7-1.el7.elrepo                            elrepo-kernel
安装最新版：
yum --enablerepo=elrepo-kernel install kernel-ml kernel-ml-devel –y
安装完成后reboot
更改内核顺序：
grub2-set-default  0 && grub2-mkconfig -o /etc/grub2.cfg && grubby --args="user_namespace.enable=1" --update-kernel="$(grubby --default-kernel)" && reboot
开机后查看内核
[appadmin@k8s-node01 ~]$ uname -a
Linux k8s-node01 5.7.7-1.el7.elrepo.x86_64 #1 SMP Wed Jul 1 11:53:16 EDT 2020 x86_64 x86_64 x86_64 GNU/Linux
CentOS 8按需升级:
可以采用dnf升级，也可使用上述同样步骤升级（使用上述步骤注意elrepo-release-8.1版本）
rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
dnf install https://www.elrepo.org/elrepo-release-8.1-1.el8.elrepo.noarch.rpm
dnf --disablerepo=\* --enablerepo=elrepo -y install kernel-ml kernel-ml-devel
grubby --default-kernel && reboot

1.6 docker 安装准备

安装docker-19.03 
 yum install docker-ce-19.03* docker-ce-cli-19.03* -y 
service docker start 
chkconfig docker on 
配置docker的cgroup:
cat > /etc/docker/daemon.json <<EOF
{
  "exec-opts": ["native.cgroupdriver=systemd"]
}
EOF

二： 安装 kubernetes 准备

下载地址github
   k8s github : https://github.com/kubernetes/kubernetes/
安装k8s1.20.8
tar -zxvf kubernetes-server-linux-v1.20.8-amd64.tar --strip-components=3 -C /usr/local/bin kubernetes/server/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy}

配置etcd
 tar -zxvf etcd-v3.4.14-linux-amd64.tar.gz --strip-components=1 -C /usr/local/bin etcd-v3.4.14-linux-amd64/{etcd,etcdctl}

vim k8s.sh
-------
MasterNodes='k8s-master02 k8s-master03'
WorkNodes='k8s-node01 k8s-node02 k8s-node03'
for NODE in $MasterNodes; do echo $NODE; scp /usr/local/bin/kube{let,ctl,-apiserver,-controller-manager,-scheduler,-proxy} $NODE:/usr/local/bin/; scp /usr/local/bin/etcd* $NODE:/usr/local/bin/; done
for NODE in $WorkNodes; do     scp /usr/local/bin/kube{let,-proxy} $NODE:/usr/local/bin/ ; done
-------
. ./k8s.sh

CNI安装，下载CNI组件
wget https://github.com/containernetworking/plugins/releases/download/v0.8.5/cni-plugins-linux-amd64-v0.8.5.tgz
所有节点：
mkdir -p /opt/cni/bin
解压cni并发送至其他节点
tar -zxf cni-plugins-linux-amd64-v0.8.5.tgz -C /opt/cni/bin
vim cni.sh 
--------
MasterNodes='k8s-master02 k8s-master03'
WorkNodes='k8s-node01 k8s-node02 k8s-node03'
for NODE in $MasterNodes; do ssh $NODE 'mkdir -p /opt/cni/bin'; scp /opt/cni/bin/* $NODE:/opt/cni/bin/; done
for NODE in $WorkNodes; do ssh $NODE 'mkdir -p /opt/cni/bin'; scp /opt/cni/bin/* $NODE:/opt/cni/bin/; done
---------
. ./cni.sh