Navigation
This version of the manual is no longer supported.
  • Replication >
  • Deploy a Geographically Distributed Replica Set

Deploy a Geographically Distributed Replica Set

This tutorial outlines the process for deploying a replica set with members in multiple locations. The tutorial addresses three-member sets, four-member sets, and sets with more than four members.

For appropriate background, see Replica Set Fundamental Concepts and Replica Set Architectures and Deployment Patterns. For related tutorials, see Deploy a Replica Set and Add Members to a Replica Set.

Overview

While replica sets provide basic protection against single-instance failure, when all of the members of a replica set reside in a single facility, the replica set is still susceptible to some classes of errors in that facility including power outages, networking distortions, and natural disasters. To protect against these classes of failures, deploy a replica set with one or more members in a geographically distinct facility or data center.

Requirements

If possible, use an odd number of data centers, and choose a distribution of members that maximizes the likelihood that even with a loss of a data center, the remaining replica set members can form a majority or at minimum, provide a copy of your data.

If you deploy a replica set with an uneven number of members, add an arbiter.

For all configurations in this tutorial, deploy each replica set member on a separate system. Although you may deploy more than one replica set member on a single system, doing so reduces the redundancy and capacity of the replica set. Such deployments are typically for testing purposes and beyond the scope of this tutorial.

Procedures

Deploy a Distributed Three-Member Replica Set

A geographically distributed three-member deployment has the following features:

  • Each member of the replica set resides on its own machine, and the MongoDB processes all bind to port 27017, which is the standard MongoDB port.

  • Each member of the replica set must be accessible by way of resolvable DNS or hostnames in the following scheme:

    • mongodb0.example.net
    • mongodb1.example.net
    • mongodb2.example.net

    Configure DNS names appropriately, or set up your systems’ /etc/hosts file to reflect this configuration.

  • Ensure that network traffic can pass between all members in the network securely and efficiently. Consider the following:

    • Establish a virtual private network between the systems in Site A and Site B to encrypt all traffic between the sites and remains private. Ensure that your network topology routes all traffic between members within a single site over the local area network.

    • Configure authentication using auth and keyFile, so that only servers and process with authentication can connect to the replica set.

    • Configure networking and firewall rules so that only traffic (incoming and outgoing packets) on the default MongoDB port (e.g. 27017) from within your deployment.

      See also

      For more information on security and firewalls, see Security Considerations for Replica Sets.

  • Specify run-time configuration on each system in a configuration file stored in /etc/mongodb.conf or in a related location. Do not specify run-time configuration through command line options.

    For each MongoDB instance, use the following configuration, with values set appropriate to your systems:

    port = 27017
    
    bind_ip = 10.8.0.10
    
    dbpath = /srv/mongodb/
    
    fork = true
    
    replSet = rs0/mongodb0.example.net,mongodb1.example.net,mongodb2.example.net
    

    Modify bind_ip to reflect a secure interface on your system that is able to access all other members of the set and that is accessible to all other members of the replica set. The DNS or host names need to point and resolve to this IP address. Configure network rules or a virtual private network (i.e. “VPN”) to permit this access.

    Note

    The portion of the replSet following the / provides a “seed list” of known members of the replica set. mongod uses this list to fetch configuration changes following restarts. It is acceptable to omit this section entirely, and have the replSet option resemble:

    replSet = rs0
    

    For more documentation on the above run time configurations, as well as additional configuration options, see Configuration File Options.

To deploy a geographically distributed three-member set:

  1. On each system start the mongod process by issuing a command similar to following:

    mongod --config /etc/mongodb.conf
    

    Note

    In production deployments you likely want to use and configure a control script to manage this process based on this command. Control scripts are beyond the scope of this document.

  2. Open a mongo shell connected to one of the mongod instances:

    mongo
    
  3. Use the rs.initiate() method on one member to initiate a replica set consisting of the current member and using the default configuration:

    rs.initiate()
    
  4. Display the current replica configuration:

    rs.conf()
    
  5. Add the remaining members to the replica set by issuing a sequence of commands similar to the following. The example commands assume the current primary is mongodb0.example.net:

    rs.add("mongodb1.example.net")
    rs.add("mongodb2.example.net")
    
  6. Optional. Configure the member eligibility for becoming primary. In some cases, you may prefer that the members in one data center be elected primary before the members in the other data centers.

    1. Issue the following command to determine the _id value for mongodb2.example.net:

      rs.conf()
      
    2. In the members array, find the array index of the member to configure. The example in the next step assumes this value is 2.

    3. In the mongo shell connected to the replica set’s primary, issue a command sequence similar to the following:

      cfg = rs.conf()
      cfg.members[2].priority = 0
      rs.reconfig(cfg)
      

      Note

      In some situations, the rs.reconfig() shell method can force the current primary to step down and causes an election. When the primary steps down, all clients will disconnect. This is the intended behavior. While, this typically takes 10-20 seconds, attempt to make these changes during scheduled maintenance periods.

    After these commands return you have a geographically distributed three-member replica set.

  7. To check the status of your replica set, issue rs.status().

See also

The documentation of the following shell functions for more information:

Deploy a Distributed Four-Member Replica Set

A geographically distributed four-member deployment has the following features:

  • Each member of the replica set, except for the arbiter (see below), resides on its own machine, and the MongoDB processes all bind to port 27017, which is the standard MongoDB port.

  • Each member of the replica set must be accessible by way of resolvable DNS or hostnames in the following scheme:

    • mongodb0.example.net
    • mongodb1.example.net
    • mongodb2.example.net
    • mongodb3.example.net

    Configure DNS names appropriately, or set up your systems’ /etc/host file to reflect this configuration.

  • One host (e.g. mongodb3.example.net) will be an arbiter and can run on a system that is also used for an application server or some other shared purpose.

  • There are three possible architectures for this replica set:

    • Two members in Site A, two secondary-only members in Site B, and an arbiter in Site A.
    • Three members in Site A and one secondary-only member in Site B.
    • Two members in Site A, one secondary-only member in Site B, one secondary-only member in Site C, and an arbiter in site A.

    In most cases the first architecture is preferable because it is the least complex.

  • Ensure that network traffic can pass between all members in the network securely and efficiently. Consider the following:

    • Establish a virtual private network between the systems in Site A and Site B (and Site C if it exists) to encrypt all traffic between the sites and remains private. Ensure that your network topology routes all traffic between members within a single site over the local area network.

    • Configure authentication using auth and keyFile, so that only servers and process with authentication can connect to the replica set.

    • Configure networking and firewall rules so that only traffic (incoming and outgoing packets) on the default MongoDB port (e.g. 27017) from within your deployment.

      See also

      For more information on security and firewalls, see Security Considerations for Replica Sets.

  • Specify run-time configuration on each system in a configuration file stored in /etc/mongodb.conf or in a related location. Do not specify run-time configuration through command line options.

    For each MongoDB instance, use the following configuration, with values set appropriate to your systems:

    port = 27017
    
    bind_ip = 10.8.0.10
    
    dbpath = /srv/mongodb/
    
    fork = true
    
    replSet = rs0/mongodb0.example.net,mongodb1.example.net,mongodb2.example.net,mongodb3.example.net
    

    Modify bind_ip to reflect a secure interface on your system that is able to access all other members of the set and that is accessible to all other members of the replica set. The DNS or host names need to point and resolve to this IP address. Configure network rules or a virtual private network (i.e. “VPN”) to permit this access.

    Note

    The portion of the replSet following the / provides a “seed list” of known members of the replica set. mongod uses this list to fetch configuration changes following restarts. It is acceptable to omit this section entirely, and have the replSet option resemble:

    replSet = rs0
    

    For more documentation on the above run time configurations, as well as additional configuration options, see doc:/reference/configuration-options.

To deploy a geographically distributed four-member set:

  1. On each system start the mongod process by issuing a command similar to following:

    mongod --config /etc/mongodb.conf
    

    Note

    In production deployments you likely want to use and configure a control script to manage this process based on this command. Control scripts are beyond the scope of this document.

  2. Open a mongo shell connected to this host:

    mongo
    
  3. Use rs.initiate() to initiate a replica set consisting of the current member and using the default configuration:

    rs.initiate()
    
  4. Display the current replica configuration:

    rs.conf()
    
  5. Add the remaining members to the replica set by issuing a sequence of commands similar to the following. The example commands assume the current primary is mongodb0.example.net:

    rs.add("mongodb1.example.net")
    rs.add("mongodb2.example.net")
    rs.add("mongodb3.example.net")
    
  6. In the same shell session, issue the following command to add the arbiter (e.g. mongodb4.example.net):

    rs.addArb("mongodb4.example.net")
    
  7. Optional. Configure the member eligibility for becoming primary. In some cases, you may prefer that the members in one data center be elected primary before the members in the other data centers.

    1. Issue the following command to determine the _id value for the member:

      rs.conf()
      
    2. In the members array, find the array index of the member to configure. The example in the next step assumes this value is 2.

    3. In the mongo shell connected to the replica set’s primary, issue a command sequence similar to the following:

      cfg = rs.conf()
      cfg.members[2].priority = 0
      rs.reconfig(cfg)
      

      Note

      In some situations, the rs.reconfig() shell method can force the current primary to step down and causes an election. When the primary steps down, all clients will disconnect. This is the intended behavior. While, this typically takes 10-20 seconds, attempt to make these changes during scheduled maintenance periods.

    After these commands return you have a geographically distributed four-member replica set.

  8. To check the status of your replica set, issue rs.status().

See also

The documentation of the following shell functions for more information:

Deploy a Distributed Set with More than Four Members

The above procedures detail the steps necessary for deploying a geographically redundant replica set. Larger replica set deployments generally follow the same steps, but have additional considerations: