- Replication >
- Replica Set High Availability >
- Rollbacks During Replica Set Failover
Rollbacks During Replica Set Failover¶
A rollback reverts write operations on a former primary when the member rejoins its replica set after a failover. A rollback is necessary only if the primary had accepted write operations that the secondaries had not successfully replicated before the primary stepped down. When the primary rejoins the set as a secondary, it reverts, or “rolls back,” its write operations to maintain database consistency with the other members.
MongoDB attempts to avoid rollbacks, which should be rare. When a rollback does occur, it is often the result of a network partition. Secondaries that can not keep up with the throughput of operations on the former primary, increase the size and impact of the rollback.
A rollback does not occur if the write operations replicate to another member of the replica set before the primary steps down and if that member remains available and accessible to a majority of the replica set.
Collect Rollback Data¶
Configure Rollback Data¶
Starting in version 4.0, MongoDB adds the parameter
createRollbackDataFiles
to control whether or not rollback
files are created during rollbacks.
Rollback Data¶
By default, when a rollback occurs, MongoDB writes the rollback data to
BSON files. For each collection whose data is rolled back, the
rollback files are located in a <dbpath>/rollback/<db>.<collection>
directory and have filenames of the form: [1]
For example, if data for the collection comments
in the
reporting
database rolled back:
where <dbpath>
is the mongod
’s dbPath
.
If the operation to roll back is a collection drop or a document deletion, the rollback of the collection drop or document deletion is not written to the rollback data directory.
Read Rollback Data¶
To read the contents of the rollback files, use bsondump. Based on the content and the knowledge of their applications, administrators can decide the next course of action to take.
[1] | In previous versions, rollback files are located directly under the
<dbpath>/rollback directory with the filenames of the form
<db>.<collection>.<timestamp>.bson . |
Avoid Replica Set Rollbacks¶
For replica sets, the default write concern {w: 1} only provides acknowledgement of write
operations on the primary. With the default write concern, data may be
rolled back if the primary steps down before the write operations have
replicated to any of the secondaries. This includes data written in
multi-document transactions that commit
using "w: 1"
write concern.
To prevent rollbacks of data that have been acknowledged to the client, run all voting members with journaling enabled and use w: majority write concern to guarantee that the write operations propagate to a majority of the replica set nodes before returning with acknowledgement to the issuing client.
With writeConcernMajorityJournalDefault
set to false
,
MongoDB does not wait for w: "majority"
writes to be written to the on-disk journal before acknowledging the
writes. As such, majority
write operations could
possibly roll back in the event of a transient loss (e.g. crash and
restart) of a majority of nodes in a given replica set.
Note
Regardless of write concern, other clients using
"local"
or"available"
readConcern can see the result of a write operation before the write operation is acknowledged to the issuing client.For operations in a multi-document transaction, the data changes made in the transaction are not visible outside the transaction until a transaction commits. However, other clients can see the result at the time of the transaction commit before the commit operation is acknowledged to the issuing client.
Clients using
"local"
or"available"
readConcern can read data which may be subsequently rolled back during replica set failovers.
Rollback Considerations¶
Background Index Builds¶
Changed in version 4.0.
Starting in version 4.0, MongoDB waits for any in-progress background index builds to finish before starting a rollback.
Size Limitations¶
Changed in version 4.0.
Starting in version 4.0, MongoDB has no limit on the amount of data that can be rolled back.
In previous versions, a mongod
instance will not
roll back more than 300 megabytes of data and requires manual
intervention if more than 300 megabytes of data need to be rolled back.
Rollback Elapsed Time Limitations¶
Starting in version 4.0, the rollback time limit defaults to 24 hours
and is configurable using the parameter
rollbackTimeLimitSecs
:
- In MongoDB 4.0.13+, the rollback time limit is calculated between the first operation after the common point and the last point in the oplog for the member to roll back.
- In MongoDB 4.0.0-4.0.12, the rollback time limit is calculated between the common point and the last point in the oplog for the member to roll back.
In MongoDB 3.6 and earlier, the rollback time limit is not configurable. For these versions, rollback is limited by the amount of data, with a maximum of 300 megabytes.
See also