ElasticSearch的水平扩容与缩容

ElasticSearch系列

一： ElasticSearch扩容

1.1 es 的扩容步骤

垂直扩容：
      纵向扩容，买更好的服务器，增加CPU,内存，在一台机器上面增加节点。1+1=2 
水平扩容：
     横向扩容，买更多的服务器，一台机器一个节点 9:9 ，11:11
在线扩容
1. 备份 数据 
2. 增加两台机器。
     192.168.100.50 、    192.168.100.51 一般扩容扩数据节点。
 3. 准备操作系统
 4. 做ssh 无秘登录
 5. 安装jdk 安装ES 软件
 6. 配置ES 参数
 7. 检查集群状态
    curl -XGET 'http://192.168.100.44:9200/_cat/health?pretty' -H 'Content-Type:application/json'
    保证集群是green 状态

8.停止外部数据写入
9.  停止分片重新分布
10. 启动新扩的节点。
11. 检查节点状态
12. 开启分片重新分布

二：es 的水平扩容

2.1 增加节点系统初始化：

cat /etc/hosts
----
192.168.100.41 rhel01.flyfish.cn
192.168.100.42 rhel02.flyfish.cn
192.168.100.43 rhel03.flyfish.cn
192.168.100.44 rhel04.flyfish.cn
192.168.100.45 rhel05.flyfish.cn
192.168.100.46 rhel06.flyfish.cn
192.168.100.47 rhel07.flyfish.cn
192.168.100.48 rhel08.flyfish.cn
192.168.100.49 rhel09.flyfish.cn
192.168.100.50 rhel10.flyfish.cn
192.168.100.51 rhel11.flyfish.cn
----
rhel10.flyfish.cn / rhel11.flyfish.cn 做为扩容节点

全部系统配置：
系统语言：
echo "export LANG=en_US.UTF8" >> ~/.bash_profile
cat ~/.bash_profile
---
分区：
pvcreate /dev/sdb
vgcreate esdatavg /dev/sdb
lvcreate -n esdatalv -L 100000M esdatavg
mkfs.xfs /dev/esdatavg/esdatalv
---
vi /etc/fstab
---
/dev/esdatavg/esdatalv /esdb xfs defaults 0 0
---
mkdir /esdb
mount /esdb

mkdir -p /esdb/soft
groupadd -g 60001 esadmin
useradd -u 61001 -g esadmin esadmin
chown -R esadmin:esadmin /esdb
chmod -R 775 /esdb
echo "esadmin" | passwd --stdin esadmin
systemctl set-default multi-user.target
chown esadmin:esadmin -R /esdb
cat >> /etc/security/limits.conf << EOF
root soft nofile 1048576
root hard nofile 1048576
esadmin soft nproc 1048576
esadmin hard nproc 1048576
esadmin soft nofile 1048576
esadmin hard nofile 1048576
esadmin soft stack 10240
esadmin hard stack 32768
esadmin hard memlock unlimited
esadmin soft memlock unlimited
EOF
---
---
cat >> /etc/sysctl.conf << EOF
fs.aio-max-nr = 1048576
fs.file-max = 6815744
net.ipv4.ip_local_port_range = 1024 65535
net.ipv4.tcp_mem = 786432 2097152 3145728
net.ipv4.tcp_rmem = 4096 4096 16777216
net.ipv4.tcp_wmem = 4096 4096 16777216
net.ipv4.tcp_tw_reuse = 1
net.ipv4.tcp_tw_recycle = 1
vm.swappiness=1
vm.min_free_kbytes=204800
vm.max_map_count=2048000
kernel.pid_max=819200
vm.zone_reclaim_mode=0
#vm.nr_hugepages = 0
EOF
sysctl -p 

ln -sf /usr/share/zoneinfo/Asia/Shanghai /etc/localtime
hwclock
echo "SELINUX=disabled" > /etc/selinux/config
echo "#SELINUXTYPE=targeted " >> /etc/selinux/config
cat /etc/selinux/config
setenforce 0
systemctl stop firewalld.service
systemctl disable firewalld.service
cat >> /home/esadmin/.vimrc << EOF
map <F10>:set paste<CR>
map <F11>:set nopaste<CR>
EOF
shutdown -r now

做所有主机的无密钥登录：
  rm -rf /home/esadmin/.ssh/* 
./fgssh -user esadmin -hosts "rhel01.flyfish.cn rhel02.flyfish.cn rhel03.flyfish.cn rhel04.flyfish.cn rhel05.flyfish.cn rhel06.flyfish.cn rhel07.flyfish.cn rhel08.flyfish.cnrhel09.flyfish.cn rhel10.flyfish.cn rhel11.flyfish.cn" -advanced -exverify -confirm
chmod 600 /home/esadmin/.ssh/config

配置jdk
   su - esadmin
cd /esdb/
  scp -r /esdb/jdk-14.0.2 esadmin@rhel10.flyfish.cn:/esdb/
  scp -r /esdb/jdk-14.0.2 esadmin@rhel11.flyfish.cn:/esdb/

su - esadmin
cat >> ~/.bash_profile << EOF
export JAVA_HOME=/esdb/jdk-14.0.2
export PATH=/esdb/jdk-14.0.2/bin:$PATH
export LANG=en_US.UTF8
EOF
source ~/.bash_profile
java -version

同步esdb 安装目录
scp -r /esdb/ esadmin@rhel10.flyfish.cn:/
scp -r /esdb/ esadmin@rhel11.flyfish.cn:/
rm -rf /esdb/esdata/esdata9200/*
rm -rf /esdb/eslog/eslog9200/* 

修改es的yaml
rhel10.flyfish.cn:
 vim  /esdb/esapp/esapp9200/config/elasticsearch.yml
----
cluster.name: flyfishescluster1
node.name: 192.168.100.50
network.host: 192.168.100.50
http.port: 9200
transport.tcp.port: 9300
path.data: /esdb/esdata/esdata9200
path.logs: /esdb/eslog/eslog9200
node.master: false
node.data: true
cluster.remote.connect: false
http.cors.enabled: true
http.cors.allow-origin: "*"
bootstrap.memory_lock: true
bootstrap.system_call_filter: false
cluster.initial_master_nodes: ["192.168.100.44:9300","192.168.100.45:9300","192.168.100.46:9300"]
discovery.seed_hosts: ["192.168.100.44:9300", "192.168.100.45:9300", "192.168.100.46:9300"]
discovery.zen.fd.ping_timeout: 60s
discovery.zen.fd.ping_retries: 5
----
rhel11.flyfish.cn:
 vim /esdb/esapp/esapp9200/config/elasticsearch.yml
----
cluster.name: flyfishescluster1
node.name: 192.168.100.51
network.host: 192.168.100.51
http.port: 9200
transport.tcp.port: 9300
path.data: /esdb/esdata/esdata9200
path.logs: /esdb/eslog/eslog9200
node.master: false
node.data: true
cluster.remote.connect: false
http.cors.enabled: true
http.cors.allow-origin: "*"
bootstrap.memory_lock: true
bootstrap.system_call_filter: false
cluster.initial_master_nodes: ["192.168.100.44:9300","192.168.100.45:9300","192.168.100.46:9300"]
discovery.seed_hosts: ["192.168.100.44:9300", "192.168.100.45:9300", "192.168.100.46:9300"]
discovery.zen.fd.ping_timeout: 60s
discovery.zen.fd.ping_retries: 5
----

curl -XGET 'http://192.168.100.41:9200/_cat/health?pretty' -H 'Content-Type:
application/json'

curl 'http://192.168.100.41:9200/_cat/shards?v'
curl '192.168.100.44:9200/_cat/nodes?v'

curl -XPUT http://192.168.100.44:9200/_cluster/settings -H 'Content-Type:application/json' -d'{"transient":{"cluster.routing.allocation.enable":"none"}}'

启动增加的节点：
      rhel10.flyfish.cn:
      /esdb/esapp/esapp9200/bin/elasticsearch 
     rhel11.flyfish.cn:
      /esdb/esapp/esapp9200/bin/elasticsearch 

后台启动：
     /esdb/esapp/esapp9200/bin/elasticsearch -d >/dev/null 2>&1 &

查看节点状态：
       curl '192.168.100.44:9200/_cat/nodes?v'

打开节点的路由平衡数据：
     curl -XPUT http://192.168.100.44:9200/_cluster/settings -H 'Content-Type:application/json' -d'{"transient":{"cluster.routing.allocation.enable":"all"}}'

查看集群分片
     curl 'http://192.168.100.41:9200/_cat/shards?v'

集群状态：
  curl -XGET 'http://192.168.100.41:9200/_cat/health?pretty' -H 'Content-Type:application/json'       

三： ES 集群的缩容

3.1 ES 的的缩容步骤

1. 监测集群状态
    curl -XGET 'http://192.168.100.41:9200/_cat/health?pretty' -H 'Content-Type:application/json'
     curl 'http://192.168.100.41:9200/_cat/shards?v'
   curl '192.168.100.44:9200/_cat/nodes?v'
   保证集群是green 状态

2. 暂停数据写入
3. 有条件就备份数据
4. 禁用集群分片平衡
curl -XPUT http://192.168.100.44:9200/_cluster/settings -H 'Content-Type:application/json' -d'{"transient":{"cluster.routing.allocation.enable":"none"}}'
5. 分片数据同步：
   curl http://192.168.100.41:9200/_flush/synced
6. 收缩节点
   一个一个操作，不能同时关闭两个节点
   先关192.168.100.50 
7. 开启集群分片平衡
curl -XPUT http://192.168.100.44:9200/_cluster/settings -H 'Content-Type:application/json' -d'{"transient":{"cluster.routing.allocation.enable":"all"}}'
8 查看集群状态：
     curl -XGET 'http://192.168.100.41:9200/_cat/health?pretty' -H 'Content-Type:application/json'
     curl 'http://192.168.100.41:9200/_cat/shards?v'
   curl '192.168.100.44:9200/_cat/nodes?v'
   保证集群是green 状态
 9.继续收缩其他节点。

第二种：排除集群节点方法：
   PUT _cluster/settings
{
"transient" : {
"cluster.routing.allocation.exclude._ip" : "192.168.100.50"
}
}

下降：rhel10.flyfish.cn 
 先检查集群状态：
   curl -XGET 'http://192.168.100.41:9200/_cat/health?pretty' -H 'Content-Type:application/json'
     curl 'http://192.168.100.41:9200/_cat/shards?v'
   curl '192.168.100.44:9200/_cat/nodes?v'

禁用数据平衡
   curl -XPUT http://192.168.100.44:9200/_cluster/settings -H 'Content-Type:application/json' -d'{"transient":{"cluster.routing.allocation.enable":"none"}}'

停掉 rhel10.flyfish.cn 上面的 es 
  kill -9 2993

查看集群状态：

重新平衡：

查看集群状态

四： ES 的升级

ES7+ 升级很方便
下载一个新版本
 copy 配置文件
 启动
 1.做好备份，
 2. 下载新的版本安装
 3. 集群停止分片平衡
 4. copy 配置文件启动
 5. 启动ES 分片平衡

停止数据平衡：
curl -XPUT http://192.168.100.41:9200/_cluster/settings -H 'Content-Type:application/json' -d'{"transient":{"cluster.routing.allocation.enable":"none"}}'

停掉集群：
  ./es-stop.sh 

cd /esdb/soft/
tar -zxvf elasticsearch-7.9.1-linux-x86_64.tar.gz
---

备份原有所有主机：
---
cd /esdb/esapp
mv esapp9200/ esapp9200-7.9.0
---

cd /esdb/esapp/ 
mkdir esapp9200

cd /esdb/soft/elasticsearch-7.9.1
cp -r * /esdb/esapp/esapp9200/
scp -r /esdb/esapp/esapp9200/ esadmin@rhel02.flyfish.cn:/esdb/esapp/
scp -r /esdb/esapp/esapp9200/ esadmin@rhel03.flyfish.cn:/esdb/esapp/

复制每台机器的配置文件覆盖
cd /esdb/esapp/esapp9200-7.9.0/config
cp -p jvm.options ../../esapp9200/config/
cp -p elasticsearch.yml ../../esapp9200/config/

所有机器更改变量：
vi /esdb/esapp/esapp9200/bin/elasticsearch-env
---
export JAVA_HOME=/esdb/jdk-14.0.2
export PATH=$JAVA_HOME/bin:$PATH
---

启动es:
  /esdb/esapp/esapp9200/bin/elasticsearch    

查看es的装态：
      curl '192.168.100.41:9200/_cat/nodes?v'
      curl 'http://192.168.100.41:9200/_cat/shards?v'
打开路由平衡：
      curl -XPUT http://192.168.100.41:9200/_cluster/settings -H 'Content-Type:application/json' -d'{"transient":{"cluster.routing.allocation.enable":"all"}}'

查看数据：
      curl -XPOST 'http://192.168.100.41:9200/itpuxdb/_search?pretty' -H 'Content-Type:application/json' 

集群状态：
curl http://192.168.100.41:9200/_cat/health?pretty -H 'Content-Type: application/json'