@mritd
2017-10-10T08:00:14.000000Z
字数 3169
阅读 1767
Kubernetes
Kubernetes 1.8 发布已经好几天,1.8 对于 kube-proxy 组件增加了 ipvs 支持,以下记录一下 kube-proxy ipvs 开启教程
目前测试为 5 台虚拟机,CentOS 系统,etcd、kubernetes 全部采用 rpm 安装,使用 systemd 来做管理,网络组件采用 calico,Master 实现了 HA;基本环境如下
IP | 组件 |
---|---|
10.10.1.5 | Master、Node、etcd |
10.10.1.6 | Master、Node、etcd |
10.10.1.7 | Master、Node、etcd |
10.10.1.8 | Node |
10.10.1.9 | Node |
之所以把这个单独写一个标题是因为坑有点多,为了避免下面出现问题,先说一下注意事项:
如果对 SELinux 玩的不溜的朋友,我建议先关闭 SELinux,关闭方法如下
# 编辑 /etc/selinux/config 文件;确保 SELINUX=disabled
docker1.node ➜ ~ cat /etc/selinux/config
# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
# enforcing - SELinux security policy is enforced.
# permissive - SELinux prints warnings instead of enforcing.
# disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three two values:
# targeted - Targeted processes are protected,
# minimum - Modification of targeted policy. Only selected processes are protected.
# mls - Multi Level Security protection.
SELINUXTYPE=targeted
然后重启机器并验证
docker1.node ➜ ~ sestatus
SELinux status: disabled
搭建时尽量关闭防火墙,如果你玩的很溜,那么请在测试没问题后再开启防火墙
systemctl stop firewalld
systemctl disable firewalld
确保内核已经开启如下参数,或者说确保 /etc/sysctl.conf
有如下配置
docker1.node ➜ ~ cat /etc/sysctl.conf
# sysctl settings are defined through files in
# /usr/lib/sysctl.d/, /run/sysctl.d/, and /etc/sysctl.d/.
#
# Vendors settings live in /usr/lib/sysctl.d/.
# To override a whole file, create a new file with the same in
# /etc/sysctl.d/ and put new settings there. To override
# only specific settings, add a file with a lexically later
# name in /etc/sysctl.d/ and put new settings there.
#
# For more information, see sysctl.conf(5) and sysctl.d(5).
net.ipv4.ip_forward=1
net.bridge.bridge-nf-call-iptables=1
net.bridge.bridge-nf-call-ip6tables=1
然后执行 sysctl -p
使之生效
docker1.node ➜ ~ sysctl -p
net.ipv4.ip_forward = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
由于 ipvs 已经加入到内核主干,所以需要内核模块支持,请确保内核已经加载了相应模块;如不确定,执行以下脚本,以确保内核加载相应模块,否则会出现 failed to load kernel modules: [ip_vs_rr ip_vs_sh ip_vs_wrr]
错误
cat > /etc/sysconfig/modules/ipvs.modules <<EOF
#!/bin/bash
ipvs_modules="ip_vs ip_vs_lc ip_vs_wlc ip_vs_rr ip_vs_wrr ip_vs_lblc ip_vs_lblcr ip_vs_dh ip_vs_sh ip_vs_fo ip_vs_nq ip_vs_sed ip_vs_ftp nf_conntrack_ipv4"
for kernel_module in \${ipvs_modules}; do
/sbin/modinfo -F filename \${kernel_module} > /dev/null 2>&1
if [ $? -eq 0 ]; then
/sbin/modprobe \${kernel_module}
fi
done
EOF
chmod 755 /etc/sysconfig/modules/ipvs.modules && bash /etc/sysconfig/modules/ipvs.modules && lsmod | grep ip_vs
执行后应该如下图所示,如果 lsmod | grep ip_vs
并未出现 ip_vs_rr
等模块那么请更换内核(一般不会,2.6 以后 ipvs 好像已经就合并进主干了)
修改 /etc/kubernetes/proxy
配置如下
###
# kubernetes proxy config
# default config should be adequate
# Add your own!
KUBE_PROXY_ARGS="--bind-address=10.10.1.8 \
--hostname-override=docker4.node \
--feature-gates=SupportIPVSProxyMode=true \
--proxy-mode=ipvs \
--ipvs-min-sync-period=5s \
--ipvs-sync-period=5s \
--ipvs-scheduler=rr \
--kubeconfig=/etc/kubernetes/kube-proxy.kubeconfig \
--cluster-cidr=10.254.0.0/16"
启用 ipvs 后与 1.7 版本的配置差异如下:
--feature-gates=SupportIPVSProxyMode=true
选项,用于告诉 kube-proxy 开启 ipvs 支持,因为目前 ipvs 并未稳定ipvs-min-sync-period
、--ipvs-sync-period
、--ipvs-scheduler
三个参数用于调整 ipvs,具体参数值请自行查阅 ipvs 文档修改完成后,重启 kube-proxy 使其生效
systemctl daemon-reload
systemctl restart kube-proxy
重启后日志中应该能看到如下输出,不应该有其他提示 ipvs 的错误信息出现
同时使用 ipvsadm 命令应该能看到相应的 service 的 ipvs 规则(ipvsadm 自己安装一下)
然后进入 Pod 测试