- Replication >
- Replica Set Tutorials >
- Replica Set Maintenance Tutorials >
- Reconfigure a Replica Set with Unavailable Members
Reconfigure a Replica Set with Unavailable Members¶
To reconfigure a replica set when a minority of
members are unavailable, use the rs.reconfig()
operation on
the current primary, following the example in the
Replica Set Reconfiguration Procedure.
This document provides the following options for re-configuring a replica set when a majority of members are not accessible:
You may need to use one of these procedures, for example, in a geographically distributed replica set, where no local group of members can reach a majority. See Replica Set Elections for more information on this situation.
Reconfigure by Forcing the Reconfiguration¶
Changed in version 2.0.
This procedure lets you recover while a majority of replica set
members are down or unreachable. You connect to any surviving member and
use the force
option to the rs.reconfig()
method.
The force
option forces a new configuration onto the member. Use this procedure only to
recover from catastrophic interruptions. Do not use force
every
time you reconfigure. Also, do not use the force
option in any automatic
scripts and do not use force
when there is still a primary.
To force reconfiguration:
Back up a surviving member.
Connect to a surviving member and save the current configuration. Consider the following example commands for saving the configuration:
cfg = rs.conf() printjson(cfg)
On the same member, remove the down and unreachable members of the replica set from the
members
array by setting the array equal to the surviving members alone. Consider the following example, which uses thecfg
variable created in the previous step:cfg.members = [cfg.members[0] , cfg.members[4] , cfg.members[7]]
On the same member, reconfigure the set by using the
rs.reconfig()
command with theforce
option set totrue
:rs.reconfig(cfg, {force : true})
This operation forces the secondary to use the new configuration. The configuration is then propagated to all the surviving members listed in the
members
array. The replica set then elects a new primary.Note
When you use
force : true
, the version number in the replica set configuration increases significantly, by tens or hundreds of thousands. This is normal and designed to prevent set version collisions if you accidentally force re-configurations on both sides of a network partition and then the network partitioning ends.If the failure or partition was only temporary, shut down or decommission the removed members as soon as possible.
Reconfigure by Replacing the Replica Set¶
Use the following procedure only for versions of MongoDB prior to version 2.0. If you’re running MongoDB 2.0 or later, use the above procedure, Reconfigure by Forcing the Reconfiguration.
These procedures are for situations where a majority of the
replica set members are down or unreachable. If a majority is
running, then skip these procedures and instead use the
rs.reconfig()
command according to the examples in
Example Reconfiguration Operations.
If you run a pre-2.0 version and a majority of your replica set is down, you have the two options described here. Both involve replacing the replica set.
Reconfigure by Turning Off Replication¶
This option replaces the replica set with a standalone server.
Stop the surviving
mongod
instances. To ensure a clean shutdown, use an existing control script or use thedb.shutdownServer()
method.For example, to use the
db.shutdownServer()
method, connect to the server using themongo
shell and issue the following sequence of commands:use admin db.shutdownServer()
Create a backup of the data directory (i.e.
dbpath
) of the surviving members of the set.Optional
If you have a backup of the database you may instead remove this data.
Restart one of the
mongod
instances without the--replSet
parameter.The data is now accessible and provided by a single server that is not a replica set member. Clients can use this server for both reads and writes.
When possible, re-deploy a replica set to provide redundancy and to protect your deployment from operational interruption.
Reconfigure by “Breaking the Mirror”¶
This option selects a surviving replica set member to be the
new primary and to “seed” a new replica set. In the following
procedure, the new primary is db0.example.net
. MongoDB copies the
data from db0.example.net
to all the other members.
Stop the surviving
mongod
instances. To ensure a clean shutdown, use an existing control script or use thedb.shutdownServer()
method.For example, to use the
db.shutdownServer()
method, connect to the server using themongo
shell and issue the following sequence of commands:use admin db.shutdownServer()
Move the data directories (i.e.
dbpath
) for all the members exceptdb0.example.net
, so that all the members exceptdb0.example.net
have empty data directories. For example:mv /data/db /data/db-old
Move the data files for
local
database (i.e.local.*
) so thatdb0.example.net
has no local database. For examplemkdir /data/local-old mv /data/db/local* /data/local-old/
Start each member of the replica set normally.
Connect to
db0.example.net
in amongo
shell and runrs.initiate()
to initiate the replica set.Add the other set members using
rs.add()
. For example, to add a member running ondb1.example.net
at port27017
, issue the following command:rs.add("db1.example.net:27017")
MongoDB performs an initial sync on the added members by copying all data from
db0.example.net
to the added members.
See also
© MongoDB, Inc 2008-present. MongoDB, Mongo, and the leaf logo are registered trademarks of MongoDB, Inc.