leofs 分布式存储

运维系列

一: leofs 介绍

1.1： leofs 介绍

软件简介:
LeoFS是一个高可靠性，最终一致性的分布式对象存储系统，主要功能如下:
支持多种协议: S3 REST NFS
支持大对象和小对象
内置缓存机制
多数据中心复制
自动运维支持
  1. 多协议 
       S3-api ， REST-api，NFS  
  2.  小文件/大文件 
       能处理超过GB级文件
  3.  多数据中心复制
       leofs 是一个具有高度扩展性，容错的分布式文件系统，无插件，leofs 集群可以被看做是一个大容量存储，它由一组松散连接的节点组成.
github地址：
https://github.com/leo-project/leofs
官网：
https://leo-project.net/leofs/

1.2： leofs 组成 部分

leofs 由3个核心应用组成，leofs_storage, leofs_gateway, leofs_manager 组成

二： leofs 安装

2.1 下载

https://leo-project.net/leofs/download.html

2.2 系统介绍

系统： centOS7.9x64
主机名：
     cat /etc/hosts 
----
192.168.100.11 flyfishsrvs01
192.168.100.12 flyfishsrvs02
192.168.100.13 flyfishsrvs03
192.168.100.14 flyfishsrvs04
192.168.100.15 flyfishsrvs05
192.168.100.16 flyfishsrvs06
192.168.100.17 flyfishsrvs07
----
本次安装采用前三台虚拟机

2.3 leofs 安装

2.3.1 安装leofs包

1. 安装依赖包：
  依赖环境： 【全部节点安装】
yum install gcc gcc-c++ glibc-devel make ncurses-devel openssl-devel autoconf  libuuid-devel cmake check check-devel   wget curl git gcc* vim nc redhat-lsb -y
 yum --enablerepo=centosplus install kmod-xfs xfsprogs xfsprogs-devel -y
 rpm -ivh leofs-1.4.3-1.el7.x86_64.rpm

安装目录：
   cd /usr/local/leofs/
   ls -ld * 

2.3.2 配置leofs

配置管理节点：
192.168.100.11: manager01
cd /usr/local/leofs/1.4.3/leo_manager_0/etc
cp -p leo_manager.conf leo_manager.conf.bak
vim leo_manager.conf 
---
manager.partner = manager_1@192.168.100.11
system.dc_id = dc_1
system.cluster_id = leofs_cluster    # 集群名字
consistency.num_of_replicas = 3      # 集群的副本集
consistency.write = 1
consistency.read = 1
consistency.delete = 1
consistency.rack_aware_replicas = 0
nodename = manager_0@192.168.100.11
---

192.168.100.12 manager02:
cd /usr/local/leofs/1.4.3/leo_manager_1/etc
cp -p leo_manager.conf leo_manager.conf.bak
vim leo_manager.conf
manager.partner = manager_0@192.168.100.11 
nodename = manager_1@192.168.100.12

2.3.3 配置 存储节点

所有节点创建存储目录
mkdir /data/leofs -p 
chmod 777 -R /data/leofs 

存储节点：
192.168.100.11：
cd /usr/local/leofs/1.4.3/leo_storage/etc
cp -p leo_storage.conf leo_storage.conf.bak
vim leo_storage.conf
-----
managers = [manager_0@192.168.100.11, manager_1@192.168.100.12]
obj_containers.path = [/data/leofs]      #### 有大硬盘 挂在这个目录下面
nodename = storage_1@192.168.100.11
-----

192.168.100.12: 
cd /usr/local/leofs/1.4.3/leo_storage/etc
cp -p leo_storage.conf leo_storage.conf.bak
vim leo_storage.conf
----
managers = [manager_0@192.168.100.11, manager_1@192.168.100.12]
obj_containers.path = [/data/leofs]      #### 有大硬盘 挂在这个目录下面
nodename = storage_2@192.168.100.12
-----

192.168.100.13:
cd /usr/local/leofs/1.4.3/leo_storage/etc
cp -p leo_storage.conf leo_storage.conf.bak
vim leo_storage.conf
---
managers = [manager_0@192.168.100.11, manager_1@192.168.100.12]
obj_containers.path = [/data/leofs]      #### 有大硬盘 挂在这个目录下面
nodename = storage_3@192.168.100.13
---

2.3.4 配置leofs 的gateway 的节点：

Gateway节点：
 192.168.100.13 
cd /usr/local/leofs/1.4.3/leo_gateway/etc
cp -p leo_gateway.conf leo_gateway.conf.bak
vim leo_gateway.conf
managers = [manager_0@192.168.100.11, manager_1@192.168.100.12]
 protocol = s3
 http.port = 8080
 cache.cache_ram_capacity = 268435456
 cache.cache_disc_capacity = 0
 cache.cache_expire = 300
 cache.cache_max_content_len = 1048576
 nodename = gateway_01@192.168.100.13
 cache.cache_disc_dir_data = ./cache/data
 cache.cache_disc_dir_journal = ./cache/journal

2.3.5 启动leofs:

Manager-master：
192.168.100.11 
/usr/local/leofs/1.4.3/leo_manager_0/bin/leo_manager start
/usr/local/leofs/1.4.3/leo_manager_0/bin/leo_manager ping 
Manager-slave :
192.168.100.12
/usr/local/leofs/1.4.3/leo_manager_1/bin/leo_manager start
/usr/local/leofs/1.4.3/leo_manager_1/bin/leo_manager ping 

Storage nodes (全部节点：)
/usr/local/leofs/1.4.3/leo_storage/bin/leo_storage start
/usr/local/leofs/1.4.3/leo_storage/bin/leo_storage ping 

检查会在 /data/leofs 生成文件：

/usr/local/leofs/1.4.3/leofs-adm status 

启动gateway
192.168.100.13：
启动gateway：
Gateway(s)
192.168.100.13：
  /usr/local/leofs/1.4.3/leo_gateway/bin/leo_gateway start
  /usr/local/leofs/1.4.3/leo_gateway/bin/leo_gateway ping 

存储驱动启动：
192.168.100.11 主节点：
  /usr/local/leofs/1.4.3/leofs-adm start 

  /usr/local/leofs/1.4.3/leofs-adm status

2.3.6 创建用户与常用命令：

创建使用用户：
开发的代码配置文件要链接这个
 cd /usr/local/leofs/1.4.3/
leofs-adm create-user flyfish 123456
---
access-key-id: 974a465ec459e68ffc1f
secret-access-key: 08eef244ef1d9e102c1a471c9c323b5da6d040b2
----
leofs-adm add-bucket admin-project 974a465ec459e68ffc1f

一些常用命令：
   用户查询 get-users
/usr/local/leofs/1.4.3/leofs-adm get-users

删除用户:
delete-user <user-id>
/usr/local/leofs/1.4.3/leofs-adm delete-user flyfish 

创建用户
create-user <user-id> <password>
/usr/local/leofs/1.4.3/leofs-adm create-user zhangyy 123456

endpoint 端点：  
/usr/local/leofs/1.4.3/leofs-adm add-endpoint 192.168.100.11
OK