Navigation

Recover a Standalone after an Unexpected Shutdown

Warning

The following procedure applies to standalone mongod instance.

Do not use this tutorial to recover a member of a replica set. Instead, you should either restore from a backup or resync from another member of the set, as described in Resync a Member of a Replica Set.

Tip

If you are running with journaling enabled, there is almost never any need to run repair since the server can use the journal files to restore the data files to a clean state automatically. However, you may need to run repair in cases where you need to recover from a disk-level data corruption.

Disk-level data corruption or missing data files can prevent mongod instance from starting, and journal files may be insufficient to recover automatically:

2018-10-24T18:05:18.248-0400 W STORAGE  [initandlisten] Detected unclean shutdown - mongod.lock is not empty.

...

2018-10-24T17:24:53.122-0400 E STORAGE  [initandlisten] Failed to get the cursor for uri: table:collection-2-6854866147293273505
2018-10-24T17:24:53.122-0400 E STORAGE  [initandlisten] This may be due to missing data files. ...

...

***aborting after fassert() failure

In such cases, your dbPath contains a non-empty mongod.lock file.

The following procedure uses mongod --repair to recover from these cases:

Warning

Only use mongod --repair (and the related operations repairDatabase command and db.repairDatabase() method) if you have no other options. The operations remove and do not save any corrupt data during the repair process.

Starting in MongoDB 4.0.3, for WiredTiger, mongod --repair:

  • Rebuilds all indexes.
  • Discards corrupt data.
  • Creates empty/stub files for missing data/metadata files.

For MMAPv1, mongod --repair:

  • Rebuilds all indexes.
  • Discards corrupt data.

Procedure

Important

Run the repair operation as the same user that normally runs the mongod process to avoid changing the permissions of the MongoDB data files.

1

Create a backup of the data files.

Create a backup copy of the data files in the --dbpath.

2

Start mongod with --repair.

To repair the data files, start the mongod instance with the --repair option.

If the mongod instance uses MMAPv1 Storage Engine, you can include the --repairpath option to specify a temporary directory used during repair. For details, see --repairpath.

Issue a command similar to the following for your standalone:

mongod --dbpath /data/db --repair

Upon completion, the dbpath should contain the repaired data files and an empty mongod.lock file. [1]

Note

For WiredTiger, if the repair fails to complete for any reason, you must restart the instance with the --repair option to complete the repair.

[1](1, 2) Generally, you should not manually remove the mongod.lock file. Instead, use the above procedure to recover the database. In dire situations, you can remove the file, start the database using the possibly corrupt files, and attempt to recover data from the database. However, it is impossible to predict the state of the database in these situations.