mongo_存储引擎浅谈、powerOf2SizeAllocations

summary_2018/07 mongodb

1、日常工作

1.1、mongodb存储引擎浅谈

1.2、mongodb分配策略：Power of 2 Sized Allocations

2、技术学习

2.1、mongodb存储引擎浅谈

• MongoDB supports multiple storage engines, as different engines perform better for specific workloads（3.0引入和mysql一样提供可拆卸的存储供用户选择，以适应不同的情况，在多节点的集群时，不同的节点可以采用不同的存储引擎）
  
  • WiredTiger Storage Engine (Default)
    
    • WiredTiger is the default storage engine starting in MongoDB 3.2

### Document Level Concurrency
1、在写操作上提供文档级别的并发控制，As a result, multiple clients can modify different documents of a collection at the same time.
2、For most read and write operations, WiredTiger uses optimistic concurrency control. WiredTiger uses only intent locks at the global, database and collection levels. When the storage engine detects conflicts between two operations, one will incur a write conflict causing MongoDB to transparently retry that operation.（在并发机制上使用乐观锁，除文档级别其余级别使用意向锁。当两个操作矛盾时，将透明的进行重试）
3、Some global operations, typically short lived operations involving multiple databases, still require a global “instance-wide” lock。（在一些全局操作上仍会使用全局范围的锁，比如删除一个数据库，仍需database lock）
### Snapshots and Checkpoints(快照和检查点)
1、WiredTiger uses MultiVersion Concurrency Control (MVCC).（事务机制）
2、MongoDB configures WiredTiger to create checkpoints (i.e. write the snapshot data to disk) at intervals of 60 seconds or 2 gigabytes of journal data.（用于意外宕机后，从最新的检查点恢复，若要恢复检查点之后的操作，使用journaling）
### Journal（日志）
WiredTiger uses a write-ahead transaction log in combination with checkpoints to ensure data durability。（保证数据的持久性）
Tips：WiredTiger automatically removes old journal files to maintain only the files needed to recover from last checkpoint.
Tips： 为了提高当前journal文件频繁的顺序写入速度，您可以将journal文件放在与数据库数据文件不同的文件系统下。
### Compression （压缩）
1、With WiredTiger, MongoDB supports compression for all collections and indexes. Compression minimizes storage use at the expense of additional CPU.（支持压缩集合和索引，但需要额外的cpu花销）
2、By default, WiredTiger uses block compression with the snappy compression library for all collections and prefix compression for all indexes.
### Memory Use
With WiredTiger, MongoDB utilizes both the WiredTiger internal cache and the filesystem cache.
by default,50% of (RAM - 1 GB), or256 MB.（if计算结果小于256）

• In-Memory Storage Engine
  
  • In-Memory Storage Engine is available in MongoDB Enterprise. Rather than storing documents on-disk, it retains them in-memory for more predictable data latencies

### Concurrency
采用文档级别的并发控制
### Memory Use
By default, the in-memory storage engine uses 50% of physical RAM minus 1 GB
### Durability（持久化）
### Transactions（事务）
Multi-document transactions are not available for deployments with members that use in-memory storage engine.（不支持多文档事务）
### Deployment Architectures（部署架构）
1、Replica Set（副本集）
You can deploy mongod instances that use in-memory storage engine as part of a replica set. For example, as part of a three-member replica set, you could have:
// two mongod instances run with in-memory storage engine.
// one mongod instance run with WiredTiger storage engine. Configure the WiredTiger member as a hidden member (i.e. hidden: true and priority: 0).
在这种情况，只有使用内存引擎的实例才能成为主要实例，即使它们都崩溃重启，也可以从运行WiredTiger的成员同步。
2、Sharded Cluster（分片）
You can deploy mongod instances that use in-memory storage engine as part of a sharded cluster.（可以为不同的分片添加Tag）

• MMAPv1 Storage Engine (Deprecated as of MongoDB 4.0)
  
  • MMAPv1 is the default storage engine for MongoDB versions 3.0 and earlier.
  • MMAPv1 is MongoDB’s original storage engine based on memory mapped files. It excels at workloads with high volume inserts, reads, and in-place updates.

### Journal
为了确保对MongoDB数据集的所有修改都持久地写入磁盘，默认情况下，MongoDB会记录对磁盘日志的所有修改。MongoDB写入日志比写入数据文件更频繁。
在MMAPv1存储引擎的默认配置中，MongoDB每隔60秒写入磁盘上的数据文件，并且大约每100毫秒写入日志文件。
### Record Storage Characteristics
所有记录都连续地位于磁盘上，当文档大于分配的记录时，MongoDB必须分配新记录。新分配要求MongoDB移动文档并更新引用文档的所有索引，这比现场更新花费更多时间并导致存储碎片。
3.0.0更改：
采用Power of 2 Sized Allocations策略分配，每个集合的大小为2的幂，包含文档自身+额外的大小，允许文档增大，同时减小了重新分配的次数。（类似于预分配）
### Record Allocation Strategies
// MongoDB supports multiple record allocation strategies that determine how mongod adds padding to a document when creating a record. （支持多种分配策略，用于决定添加新的文档时的分配策略）
Different allocation strategies support different kinds of workloads: the power of 2 allocations are more efficient for insert/update/delete workloads（适用于插入/更新/删除较多的情况）; while exact fit allocations is ideal for collections without update and delete workloads.（精确分配适用于没有更新和删除的情况）
### No Padding Allocation Strategy
3.0.0更改：
For collections whose workloads do not change the document sizes，you can disable the power of 2 allocation using the collMod command with the noPadding flag or the db.createCollection() method with the noPadding option.（对于插入/更新/删除较少的情况，可以禁用power of 2 allocation）
Tips：padding，可填充块，在这种分配策略下，每个dcoument后会有额外的padding，从而减少文档的移动和重新分配。
### Memory Use
With MMAPv1, MongoDB automatically uses all free memory on the machine as its cache（但是使用方式是动态的，如果有进程需要，它会给其它进程使用）

2.1、mongodb分配策略：Power of 2 Sized Allocations

MongoDB 3.0 uses the power of 2 sizes allocation as the default record allocation strategy for MMAPv1. With the power of 2 sizes allocation strategy, each record has a size in bytes that is a power of 2 (e.g. 32, 64, 128, 256, 512 … 2 MB). For documents larger than 2 MB, the allocation is rounded up to the nearest multiple of 2 MB.
Advantage:
1、将记录固定为固定的大小,可以有效的利用已释放的记录，以减少碎片。
2、额外的分配空间可以减少文档移动的次数（因此文档的存储是顺序的），同时也减少了索引更新的次数。