Navigation
This is an upcoming (i.e. in progress) version of the manual.
  • Release Notes >
  • Release Notes for MongoDB 4.4 (Development Series 4.3.x)

Release Notes for MongoDB 4.4 (Development Series 4.3.x)

MongoDB 4.4 Dev Series (4.3.x) Available

MongoDB 4.4 is currently in development as part of the 4.3 development series. While the 4.3-dev-series are available, these versions of MongoDB are for testing purposes only and not for production use.

Aggregation

Union All ($unionWith Stage)

MongoDB 4.4 adds the $unionWith aggregation stage, providing the ability to combines pipeline results from multiple collections into a single result set.

For details, see $unionWith.

Custom Aggregation Expressions

Starting in version 4.4, MongoDB provides the following new operators that allow users to define custom aggregation expressions:

With the addition of these new operators, you can use aggregation to write custom JavaScript expressions instead of relying on mapReduce and $where.

Note

Even before version 4.4, various map-reduce expressions could also be rewritten using other aggregation pipeline operators, such as $group, $merge, etc., without requiring custom functions.

For more information, see Map-Reduce to Aggregation Pipeline.

New Aggregation Operators

Operator Description
$accumulator Returns the result of a user-defined accumulator operator.
$binarySize Returns the size of a given string or binary data value’s content in bytes.
$bsonSize Returns the size in bytes of a given document (i.e. bsontype Object) when encoded as BSON.
$first Returns the first element in an array.
$function Defines a custom aggregation expression.
$last Returns the last element in an array.
$isNumber

Returns boolean true if the specified expression resolves to an integer, decimal, double, or long.

Returns boolean false if the expression resolves to any other BSON type, null, or a missing field

$replaceOne Replaces the first instance of a matched string in a given input.
$replaceAll Replaces all instances of a matched string in a given input.

General Aggregation Improvements

$out

Starting in MongoDB 4.4, $out can output to a collection in a different database. In earlier versions, $out can output to a collection to the same database where the aggregation is run.

$indexStats

Starting in MongoDB 4.4 (also available starting in 4.2.4), $indexStats includes the following fields in its output:

Field Description
shard Name of the shard, if applicable.
spec Index specification document
building A boolean flag that indicates if the index is currently being built.

$merge

Starting in MongoDB 4.4, $merge can output to the same collection that is being aggregated. You can also output to a collection which appears in other stages of the pipeline, such as $lookup.

Versions of MongoDB prior to 4.4 did not allow $merge to output to the same collection as the collection being aggregated.

Warning

When $merge outputs to the same collection that is being aggregated, documents may get updated multiple times or the operation may result in an infinite loop. This behavior occurs when the update performed by $merge changes the physical location of documents stored on disk. When the physical location of a document changes, $merge may view it as an entirely new document, resulting in additional updates. For more information on this behavior, see Halloween Problem.

$planCacheStats Changes

Starting in version 4.4,

Background Validation

Support for Background Validation

Starting in MongoDB 4.4, for the WiredTiger storage engine, you can run the validate command in the background or in the foreground. In earlier versions, you can only run validate in the foreground:

  • When run in background, the operation takes an intent shared lock on the collection and does not block reads and writes or DDL operations on the collection; however, the operation does not update collection count and size metrics.
  • When run in foreground, the operation takes an exclusive lock on the collection and blocks all reads and writes and DDL operations on the collection until the operation finishes. When run on a secondary, the operation can block all other operations on that secondary until it finishes.

For details, see the validate command and the db.collection.validate() helper.

maxValidateMBperSec Parameter

MongoDB 4.4 adds the maxValidateMBperSec parameter to limit the I/O and CPU usage when running background validation. See maxValidateMBperSec for details.

Removed Commands

MongoDB removes the following command(s) and mongo shell helper(s):

Removed Command Removed Helper Alternatives
cloneCollection db.cloneCollection()
planCacheListPlans PlanCache.getPlansByQuery()
planCacheListQueryShapes PlanCache.listQueryShapes()

Networking

Support for TCP Fast Open

Starting with MongoDB 4.4, mongod and mongos support TCP Fast Open (TFO) connections by default. TFO requires both the client and the mongod/mongos host machines support and enable TFO:

Windows

The following Windows operating systems support TFO:

  • Microsoft Windows Server 2016 or later.
  • Microsoft Windows 10 Update 1607 or later.
macOS
macOS 10.11 (El Capitan) and later support TFO
Linux

Linux operating systems running Linux Kernel 3.7 or later can support inbound TFO connections.

Linux operating systems running Linux Kernel 4.11 or later can support both inbound and outbound TFO connections.

Set the value of /proc/sys/net/ipv4/tcp_fastopen to enable support for inbound and/or outbound TFO connections:

  • Set to 1 to enable only outbound TFO connections
  • Set to 2 to enable only inbound TFO connections
  • Set to 3 to enable inbound and outbound TFO connections.

MongoDB 4.4 adds the following parameters for controlling TFO:

Parameter Description
tcpFastOpenServer

Default: true (Enabled)

Enables or disables support for inbound TFO connections to the mongod/mongos

tcpFastOpenClient

Default: true (Enabled)

Linux Operating System Only

Enables or disables support for outbound TFO connections from the mongod/mongos.

tcpFastOpenQueueSize

Default: 1024

Control the size of the queue of pending TFO connections.

MongoDB 4.4 adds the following counters to the output of serverStatus and db.serverStatus():

Counter Description
network.tcpFastOpen.kernelSetting

Linux only

Indicates kernel support for TFO.

network.tcpFastOpen.serverSupported Indicates operating system support for incoming TFO connections.
network.tcpFastOpen.clientSupported Indicates operating system support for outgoing TFO connections.
network.tcpFastOpen.accepted Indicates the total number of accepted incoming TFO connections to the mongod/mongos since the mongod/mongos last started.

A complete discussion of TFO is outside the scope of this documentation. For more information on TFO, start with the following external resources:

Security Improvements

x.509 Certificates Nearing Expiry Trigger Warnings

Starting in MongoDB 4.4, mongod/mongos logs a warning on connection if the presented x.509 certificate expires within 30 days of the mongod/mongos system clock. Specifically, the following connections to a mongod or mongos can trigger x.509 certificate expiry warnings:

The warning log message resembles the following:

<Timestamp> W NETWORK [connection] Peer certificate <Certificate Subject Name> expires...

Consider proactively renewing client x.509 certificates nearing expiration to ensure continued connectivity to the cluster.

MongoDB 4.4 adds the tlsX509ExpirationWarningThresholdDays parameter for controlling certificate expiration warning threshold. Set the parameter to 0 to disable the warning. For complete documentation, see tlsX509ExpirationWarningThresholdDays.

Replica Sets

Resumable Initial Sync

Starting in MongoDB 4.4, a secondary performing initial sync can attempt to resume the sync process if interrupted by a transient network error. The sync source must also run MongoDB 4.4 to support resumable initial sync. If the sync source runs MongoDB 4.2 or earlier, the secondary must restart the initial sync process as if it encountered a non-transient network error.

By default, the secondary tries to resume initial sync for 24 hours. MongoDB 4.4 adds the initialSyncTransientErrorRetryPeriodSeconds server parameter for controlling the amount of time the secondary attempts to resume initial sync. If the secondary cannot successfully resume the initial sync process during the configured time period, it selects a new healthy source from the replica set and restarts the initial synchronization process from the beginning.

Prior to MongoDB 4.4, the secondary would restart the entire initial sync process if it encountered an error during the process.

Rollback Directory

Starting in Mongo 4.4, the rollback directory for a collection is named after the collection’s UUID rather than the collection namespace; e.g.

<dbpath>/rollback/20f74796-d5ea-42f5-8c95-f79b39bad190/removed.2020-02-19T04-57-11.0.bson

For details, see Rollback Data.

Minimum Oplog Retention Period

Starting in MongoDB 4.4, you can specify the minimum number of hours to preserve an oplog entry. The mongod only removes an oplog entry if:

  • The oplog has reached the maximum configured size, and
  • The oplog entry is older than the configured number of hours based on the host system clock.

By default MongoDB does not set a minimum oplog retention period and automatically truncates the oplog starting with the oldest entries to maintain the configured maximum oplog size.

To configure the minimum oplog retention period when starting the mongod, either:

To configure the minimum oplog retention period on a running mongod, use replSetResizeOplog. Setting the minimum oplog retention period while the mongod is running overrides any values set on startup. You must update the value of the corresponding configuration file setting or command line option to persist those changes through a server restart.

Sharded Clusters

Refinable Shard Keys

Starting in 4.4, MongoDB provides the refineCollectionShardKey command. With the new command, you can refine a collection’s shard key by adding a suffix field or fields to the existing key. Refining a collection’s shard key allows for a more fine-grained data distribution and can address situations where the existing key has led to jumbo (i.e. indivisible) chunks due to insufficient cardinality.

For example, a collection’s shard key is { country: 1 }. You can change the shard key by adding a suffix postcode field to the shard key so that { country: 1, postcode: 1 } becomes the new shard key, allowing data distribution by both country and postcode fields.

To use the refineCollectionShardKey command, the sharded cluster must have feature compatibility version (fcv) of 4.4. For more information, see the refineCollectionShardKey command.

Note

After you refine the shard key, it may be that not all documents in the collection have the suffix field(s). To populate the missing shard key field(s), see Missing Shard Key.

Before refining the shard key, ensure that all or most documents in the collection have the suffix fields, if possible, to avoid having to populate the field afterwards.

In earlier versions, once you select a shard key, you cannot modify the shard key.

Missing Shard Keys

With the ability to refine a shard key with a suffix, it may be that not all documents in the collection have the suffix fields. Starting in version 4.4, documents in a sharded collection can be missing the shard key fields. In earlier versions, shard key fields must exist in every document for a sharded collection. For details, see Missing Shard Key.

Hedged Reads

To minimize latencies, mongos instances, by default, can use hedge reads. With hedged reads, the mongos instances can route read operations to multiple members per each queried shard and return results from the first respondent per shard. By default, mongos instances support using hedged reads. To turn off a mongos instance’s support for hedged reads, set the readHedgingMode parameter for the mongos.

Hedged reads are specified per operation as part of the read preference. Non-primary read preferences support hedged reads. Read preference nearest specifies hedged read by default.

For more information, see:

Hedged Read Parameters

Parameter Description
readHedgingMode Enables or disables mongos instance’s support for hedged reads.
maxTimeMSForHedgedReads Specifies the maximimum time limit (in milliseconds) for the additional read sent to hedge a read operation.

Hedged Read Option for Read Preference

To specify hedged read for a read preference, MongoDB 4.4 introduces the Hedged Read Option.

The following mongo shell methods can accept hedge options to enable hedged read for the specified read preference:

Hedged Read Metrics

The command serverStatus and its corresponding mongo shell method db.serverStatus() return hedgingMetrics.

balancerCollectionStatus Command

MongoDB 4.4 provides the command balancerCollectionStatus and the mongo shell helper method sh.balancerCollectionStatus() that return information about whether the chunks of a sharded collection are balanced (i.e. do not need to be moved) as of the time the command is run or need to be moved. With the command, users can verify that initial chunk creation and migration has finished.

Improved mongos Startup Procedure

Starting with MongoDB 4.4, mongos adds the following new default startup behavior:

  • mongos will now preload a sharded cluster’s routing table on startup, rather than doing so on-demand for the first incoming client connection.
  • mongos will now prewarm its connection pool to shard hosts on startup, rather than doing so on-demand for incoming client connections.

This behavior results in faster servicing of initial client connections after a mongos instance is started or restarted. In particular, this allows sites that employ multiple mongos instances to restart them as necessary, or add new ones, without initial client requests to those instances needing to wait on connection establishment.

Both routing table preloading and connection pool prewarming are enabled by default.

MongoDB 4.4 adds the following parameters for controlling this behavior:

Improved Routing Table Updates

Running flushRouterConfig is no longer required after executing the movePrimary or dropDatabase commands. These two commands now automatically refresh a sharded cluster’s routing table as needed when run. Manually issuing the flushRouterConfig command is still recommended in the cases described under flushRouterConfig Considerations.

General Sharded Clusters Improvements

Index Consistency Checks

Starting in MongoDB 4.4, the config server primary, by default, checks for index inconsistencies across the shards for sharded collections. The command serverStatus returns the field shardedIndexConsistency to report on index inconsistencies when run on the config server primary.

To configure the index consistency checks, MongoDB provides the following parameters:

Parameter Description
enableShardedIndexConsistencyCheck Enable or disable the index consistency checks.
shardedIndexConsistencyCheckIntervalMS The interval at which the config server’s primary checks the index consistency of sharded collections.

Concurrent removeShard Operations

Starting in MongoDB 4.4, you can have more than one removeShard operation in progress.

In earlier versions, removeShard returns an error if another removeShard operation is in progress.

Shard Key Limit

Starting in version 4.4, MongoDB removes the 512-byte limit on the shard key size. For MongoDB 4.2 and earlier, a shard key cannot exceed 512 bytes.

Transactions

Transactions Support Collection and Index Creation

Starting in MongoDB 4.4 with feature compatibility version (fcv) "4.4", you can:

  • Create collections inside of a multi-document transaction.
  • Create indexes inside of a multi-document transaction in the following scenarios:
    • The index is specified on a non-existing collection. In this case, the collection is created implicitly.
    • The index is specified on an empty collection which was created earlier in the same transaction.

For more details, see Create Collections and Indexes In a Transaction.

With fcv "4.2" or less, operations that affect the database catalog, such as creating or dropping a collection or an index, are not allowed in transactions. For details on transaction restrictions, see Restricted Operations.

Platform Support

  • MongoDB 4.4 removes support for:
    • Windows 7 / Server 2008 R2
    • Windows 8 / Server 2012
    • Windows 8.1 / Server 2012 R2

See Supported Platforms.

Drivers

New Drivers

The official MongoDB Swift driver is now available.

Tools

Migration to MongoDB Database Tools Project

Starting in MongoDB 4.4, the documentation for the following tools have been migrated to the MongoDB Database Tools project:

The MongoDB Database Tools use Apache License, Version 2.0. See mongodb/mongo-tools for the source code.

Note

For documentation on previous versions of the listed tools, reference that version of the MongoDB server manual.

Quick links to older documentation

mongoreplay Removed from MongoDB Packaging

Starting in MongoDB 4.4, mongoreplay is removed from MongoDB packaging. mongoreplay and its related documentation are migrated to the mongodb-labs github project. Projects in mongodb-labs are experimental and not officially supported by MongoDB.

Quick links to older documentation

General Improvements

Blocking Sort Limit Increased

If MongoDB cannot use an index or indexes to obtain the sort order for a given cursor.sort() operation, MongoDB must perform a blocking sort on the data. A blocking sort indicates that MongoDB must consume and process all input documents to the sort before returning results. Blocking sorts do not block concurrent operations on the collection or database.

Prior to MongoDB 4.4, MongoDB returned an error if a blocking sort operations required more than 32 megabytes of system memory. Starting in MongoDB 4.4, blocking sort operations increase the limit on system memory to use for the sort operation to 100 megabytes. For blocking sort operations which require more than 100 megabytes of system memory, MongoDB returns an error unless the query specifies cursor.allowDiskUse() (New in MongoDB 4.4).

For more information on sorting and index use, see Sort and Index Use.

allowDiskUse() Support for Queries with Large In-Memory Sorts

If MongoDB cannot use an index or indexes to obtain the sort order, MongoDB must perform a blocking sort operation on the data. A blocking sort indicates that MongoDB must consume and process all input documents to the sort before returning results. Blocking sorts do not block concurrent operations on the collection or database.

New in version 4.4.

If MongoDB requires using more than 100 megabytes of system memory for the blocking sort operation, MongoDB returns an error unless the query specifies cursor.allowDiskUse() (New in MongoDB 4.4). allowDiskUse() allows MongoDB to use temporary files on disk to store data exceeding the 100 megabyte system memory limit while processing a blocking sort operation. cursor.allowDiskUse() has no effect on sort operations resolved using an index or blocking sort operations which require less than 100 megabytes of system memory.

If using the find database command, specify the optional parameter allowDiskUse: true to allow MongoDB to use temporary files on disk to store data exceeding the 100 megabyte system memory limit while processing a blocking sort operation

For instructions on enabling allowDiskUse for queries issued through a MongoDB driver, defer to the documentation for your preferred MongoDB 4.4-compatible driver.

Collection Namespace Limit

Starting in MongoDB 4.4,

  • For featureCompatibilityVersion set to "4.4" or greater, MongoDB raises the limit on collection/view namespace to 255 bytes. For a collection or a view, the namespace includes the database name, the dot (.) separator, and the collection/view name (e.g. <database>.<collection>),
  • For featureCompatibilityVersion set to "4.2" or earlier, the maximum length of the collection/view namespace remains 120 bytes.

db.auth() Can Prompt for Password

Starting in MongoDB 4.4, the mongo shell method db.auth(<username>, <password>) prompts for the password if you do not pass in the password or the passwordPrompt() method for the <password>.

Support for $natural Sort on Views

Starting in MongoDB 4.4, you can specify a $natural sort when running a find operation against a view.

Validation Changes for $meta

Starting in MongoDB 4.4:

  • You can specify $meta in the projection document of a find command without specifying the $meta expression as part of a sort() document:

    db.collection.find(
        <query>,
        { score: { $meta: "textScore" } }
    )
    

    As a result, you can project the textScore metadata without sorting the results by their search relevance.

  • You can specify the $meta expression as part of a sort() document without specifying $meta in the projection document of a find command:

    db.collection.find(
        <query>
    ).sort( { score: { $meta: "textScore" } } )
    

    As a result, you can sort the resulting documents by their search relevance without projecting the textScore.

  • If you specify the $meta expression as part of a sort() document and the projection document of a find command, you can specify different field names for the sort() document and the projection document.

    In the following example, the score field associated with the $meta expression in the projection document contains the document’s textScore metadata. In contrast, the ignoredName field in the sort() document has an arbitrary name which does not affect the query.

    db.collection.find(
      <query>,
      { score: { $meta: "textScore" } }
    ).sort( { ignoredName: { $meta: "textScore" } } )
    

In previous versions of MongoDB, the same $meta expression, including the <projectedFieldName>, must appear in the sort() document and the find command’s projection document.

Support for Diagnostic Backtrace Generation

Starting in MongoDB 4.4, mongod and mongos processes running on Linux will now log a backtrace for each of their running threads upon receipt of a SIGUSR2 signal. This backtrace can be analyzed for diagnostic information or provided to MongoDB support as needed. This functionality is currently available only on the x86_64 architecture. For more information on using this feature, see Generate a Backtrace.

Updated ulimit Startup Warning

Starting in MongoDB 4.4, mongod will log a startup warning if a platform’s configured ulimit value for number of open files is under 64000. Previously, a warning would only be logged if this value was under 1000. See Recommended ulimit Settings for more information.

New replanReason Database Profiler Output Field

MongoDB 4.4 adds the replanReason field to database profiler output and diagnostic log messages. The replanReason field contains the reason the query system evicted a cached plan.

dbStats and collStats Output

The dbStats command and its mongo shell helper db.stats() return:

The collStats command, its mongo shell helper db.collection.stats(), and the $collStats aggregation stage return:

Delete and Hint

Starting in MongoDB 4.4, the delete command and the associated mongo shell methods db.collection.deleteOne() and db.collection.deleteMany() can accept a hint argument to specify the index to use.

Additionally, starting in MongoDB 4.4, when a findAndModify command has remove set to true, the command can accept a hint argument to specify the index to use.

See:

dropIndexes Can Abort In-Progress Index Builds

If an index specified to dropIndexes is still building, dropIndexes attempts to abort the in-progress build. Aborting an index build has the same effect as dropping the built index. Prior to MongoDB 4.4, dropIndexes would return an error if the collection had any in-progress index builds. This behavior also applies to the shell helpers db.collection.dropIndex() and db.collection.dropIndexes().

To drop a specific index out of a set of related in-progress builds, wait until the index builds complete and specify that index to dropIndexes or its shell helpers.

For more complete documentation, see:

drop() Can Abort In-Progress Index Builds

Starting in MongoDB 4.4, the db.collection.drop() method and drop command abort any in-progress index builds on the target collection before dropping the collection. Prior to MongoDB 4.4, attempting to drop a collection with in-progress index builds results in an error, and the collection is not dropped.

For replica sets or shard replica sets, aborting an index on the primary does not simultaneously abort secondary index builds. MongoDB attempts to abort the in-progress builds for the specified indexes on the primary and if successful creates an associated abort oplog entry. Secondary members with replicated in-progress builds wait for a commit or abort oplog entry from the primary before either committing or aborting the index build.

For replica sets or shard replica sets, aborting an index on the primary does not simultaneously abort secondary index builds. MongoDB attempts to abort the in-progress builds for the specified indexes on the primary and if successful creates an associated abort oplog entry. Secondary members with replicated in-progress builds wait for a commit or abort oplog entry from the primary before either committing or aborting the index build.

dropDatabase Can Abort In-Progress Index Builds

Starting in MongoDB 4.4, the db.dropDatabase() method and dropDatabase command abort any in-progress index builds on collections in the target database before dropping the database. Aborting an index build has the same effect as dropping the built index. Prior to MongoDB 4.4, attempting to drop a database that contains a collection with an in-progress index build results in an error, and the database is not dropped.

Changes Affecting Compatibility

Some changes can affect compatibility and may require user actions. For a detailed list of compatibility changes, see Compatibility Changes in MongoDB 4.4 (Development Series 4.3.x).

Upgrade Procedures

Feature Compatibility Version

To upgrade from 4.2 deployment, the 4.2 deployment must have featureCompatibilityVersion set to 4.2. To check the version:

db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )

For specific details on verifying and setting the featureCompatibilityVersion as well as information on other prerequisites/considerations for upgrades, refer to the individual upgrade instructions:

If you need guidance on upgrading to 4.4, MongoDB offers major version upgrade services to help ensure a smooth transition without interruption to your MongoDB application.

Report an Issue

To report an issue, see https://github.com/mongodb/mongo/wiki/Submit-Bug-Reports for instructions on how to file a JIRA ticket for the MongoDB server or one of the related projects.