- Frequently Asked Questions >
- FAQ: MongoDB Storage
FAQ: MongoDB Storage¶
On this page
This document addresses common questions regarding MongoDB’s storage system.
If you don’t find the answer you’re looking for, check the complete list of FAQs or post your question to the MongoDB User Mailing List.
Storage Engine Fundamentals¶
What is a storage engine?¶
A storage engine is the part of a database that is responsible for managing how data is stored on disk. Many databases support multiple storage engines, where different engines perform better for specific workloads. For example, one storage engine might offer better performance for read-heavy workloads, and another might support a higher-throughput for write operations.
What will be the default storage engine going forward?¶
MMAPv1 is the default storage engine in 3.0. With multiple storage engines, you can decide which storage engine is best for your application.
Can you mix storage engines in a replica set?¶
Yes. You can have a replica set members that use different storage engines.
When designing these multi-storage engine deployments consider the following:
- the oplog on each member may need to be sized differently to account for differences in throughput between different storage engines.
- recovery from backups may become more complex if your backup captures data files from MongoDB: you may need to maintain backups for each storage engine.
WiredTiger Storage Engine¶
Can I upgrade an existing deployment to a WiredTiger?¶
Yes. You can upgrade an existing deployment to WiredTiger while the deployment remains available by adding replica set members with the new storage engine and then removing members with the legacy storage engine. See the following sections of the Upgrade MongoDB to 3.0 for the complete procedure that you can use to upgrade an existing deployment:
How much compression does WiredTiger provide?¶
The ratio of compressed data to uncompressed data depends on your data and the compression library used. By default, collection data in WiredTiger use Snappy block compression; zlib compression is also available. Index data use prefix compression by default.
To what size should I set the WiredTiger cache?¶
The size of the cache is tunable through the
storage.wiredTiger.engineConfig.cacheSizeGB
setting. If
the cache does not have enough space to load additional data,
WiredTiger evicts pages from the cache to free up space.
Note
The storage.wiredTiger.engineConfig.cacheSizeGB
only limits the size of the WiredTiger
cache, not the total amount of memory used by mongod
.
The WiredTiger cache is only one component of the RAM used by
MongoDB. MongoDB also automatically uses all free memory on the
machine via the filesystem cache (data in the filesystem cache is
compressed).
In addition, the operating system will use any free RAM to buffer filesystem blocks.
To accommodate the additional consumers of RAM, you may have to decrease WiredTiger cache size. Avoid increasing the WiredTiger cache size above its default value.
The default WiredTiger cache size value assumes that there is a
single mongod
instance per node. If a single node
contains multiple instances, then you should decrease the setting to
accommodate the other mongod
instances.
If you run mongod
in a container (e.g. lxc
,
cgroups
, Docker, etc.) that does not have access to all of the
RAM available in a system, you must set storage.wiredTiger.engineConfig.cacheSizeGB
to a value less
than the amount of RAM available in the container. The exact amount
depends on the other processes running in the container.
To see statistics on the cache and eviction, use the
serverStatus
command. The
cache
field holds the information on
the cache and eviction.
To adjust the size of the WiredTiger cache, see
storage.wiredTiger.engineConfig.cacheSizeGB
and
--wiredTigerCacheSizeGB
.
MMAPv1 Storage Engine¶
What are memory mapped files?¶
A memory-mapped file is a file with data that the operating system
places in memory by way of the mmap()
system call. mmap()
thus
maps the file to a region of virtual memory. Memory-mapped files are
the critical piece of the MMAPv1 storage engine in MongoDB. By using memory
mapped files, MongoDB can treat the contents of its data files as if
they were in memory. This provides MongoDB with an extremely fast and
simple method for accessing and manipulating data.
How do memory mapped files work?¶
MongoDB uses memory mapped files for managing and interacting with all data.
Memory mapping assigns files to a block of virtual memory with a direct byte-for-byte correlation. MongoDB memory maps data files to memory as it accesses documents. Unaccessed data is not mapped to memory.
Once mapped, the relationship between file and memory allows MongoDB to interact with the data in the file as if it were memory.
Why are the files in my data directory larger than the data in my database?¶
The data files in your data directory, which is the /data/db
directory in default configurations, might be larger than the data set
inserted into the database. Consider the following possible causes:
Preallocated data files¶
MongoDB preallocates its data files to avoid filesystem fragmentation, and because of this, the size of these files do not necessarily reflect the size of your data.
The storage.mmapv1.smallFiles
option will reduce the
size of these files, which may be useful if you have many small databases on
disk.
The oplog
¶
If this mongod
is a member of a replica set, the data
directory includes the oplog.rs file, which is a
preallocated capped collection in the local
database.
The default allocation is approximately 5% of disk space on 64-bit installations. In most cases, you should not need to resize the oplog. See Oplog Sizing for more information.
The journal
¶
The data directory contains the journal files, which store write operations on disk before MongoDB applies them to databases. See Journaling.
Empty records¶
MongoDB maintains lists of empty records in data files as it deletes documents and collections. MongoDB can reuse this space, but will not, by default, return this space to the operating system.
To allow MongoDB to more effectively reuse the space, you can
de-fragment your data. To de-fragment, use the compact
command. The compact
requires up to 2 gigabytes of extra
disk space to run. Do not use compact
if you are
critically low on disk space. For more information on its behavior and
other considerations, see compact
.
compact
only removes fragmentation from MongoDB data files
within a collection and does not return any disk space to the operating
system. To return disk space to the operating system, see
How do I reclaim disk space?.
How do I reclaim disk space?¶
The following provides some option to consider when reclaiming disk space for a member of a replica set.
Note
You do not need to reclaim disk space for MongoDB to reuse freed space. See Empty records for information on reuse of freed space.
For a secondary member of a replica set, you can perform a resync of the member by: stopping the secondary member to resync, deleting all data and subdirectories from the member’s data directory, and restarting.
For details, see Resync a Member of a Replica Set.
What is the working set?¶
Working set represents the total body of data that the application uses in the course of normal operation. Often this is a subset of the total data size, but the specific size of the working set depends on actual moment-to-moment use of the database.
If you run a query that requires MongoDB to scan every document in a collection, the working set will expand to include every document. Depending on physical memory size, this may cause documents in the working set to “page out,” or to be removed from physical memory by the operating system. The next time MongoDB needs to access these documents, MongoDB may incur a hard page fault.
For best performance, the majority of your active set should fit in RAM.
What are page faults?¶
With the MMAPv1 storage engine, page faults can occur as MongoDB reads from or writes data to parts of its data files that are not currently located in physical memory. In contrast, operating system page faults happen when physical memory is exhausted and pages of physical memory are swapped to disk.
If there is free memory, then the operating system can find the page on disk and load it to memory directly. However, if there is no free memory, the operating system must:
- find a page in memory that is stale or no longer needed, and write the page to disk.
- read the requested page from disk and load it into memory.
This process, on an active system, can take a long time, particularly in comparison to reading a page that is already in memory.
See Page Faults for more information.
What is the difference between soft and hard page faults?¶
Page faults occur when MongoDB, with the MMAP storage engine, needs access to data that isn’t currently in active memory. A “hard” page fault refers to situations when MongoDB must access a disk to access the data. A “soft” page fault, by contrast, merely moves memory pages from one list to another, such as from an operating system file cache.
See Page Faults for more information.
Data Storage Diagnostics¶
How can I check the size of a collection?¶
To view the statistics for a collection, including the data size, use
the db.collection.stats()
method from the mongo
shell. The following example issues db.collection.stats()
for
the orders
collection:
MongoDB also provides the following methods to return specific sizes for the collection:
db.collection.dataSize()
to return data size in bytes for the collection.db.collection.storageSize()
to return allocation size in bytes, including unused space.db.collection.totalSize()
to return the data size plus the index size in bytes.db.collection.totalIndexSize()
to return the index size in bytes.
The following script prints the statistics for each database:
The following script prints the statistics for each collection in each database:
How can I check the size of indexes for a collection?¶
To view the size of the data allocated for an index, use the
db.collection.stats()
method and check the
indexSizes
field in the returned document.
How can I get information on the storage use of a database?¶
The db.stats()
method in the mongo
shell returns
the current state of the “active” database. For the description of the
returned fields, see dbStats Output.