Docs Menu

Optimize Sync Atlas Usage

On this page

  • Overview
  • History
  • Backend Compaction
  • Advanced Backend Compaction
  • Client Maximum Offline Time
  • Key Concepts
  • Client Maximum Offline Time Does Not Immediately Influence Client Resets
  • Set the Client Maximum Offline Time
  • Summary

Realm Sync uses space in your app's synced Atlas cluster to store utility data for sync. This includes a history of changes to each realm. MongoDB Realm minimizes that space usage via backend compaction, which runs automatically. You can optimize further by specifying a client maximum offline time, which enables advanced backend compaction.

The Realm backend keeps a history of changes to underlying data for each realm, similar to the MongoDB oplog. Realm uses this history to synchronize data between the backend and clients. Realm stores history in your synced Atlas cluster.

Backend compaction is a maintenance process that automatically runs for all Realm apps. During compaction, the backend optimizes a realm's history by removing unneeded changesets. The process removes any instruction whose effect is later overwritten by a newer instruction.

Example

Consider the following realm history:

Original History
CREATE table1.object1
UPDATE table1.object1.x = 4
UPDATE table1.object1.x = 10
UPDATE table1.object1.x = 42

The following history would also converge to the same state, but without unneeded interim changes:

Compacted History
CREATE table1.object1
UPDATE table1.object1.x = 42

Backend compaction:

  • reduces space usage in your Atlas cluster, since your history contains fewer instructions.
  • reduces the time and data usage of sync, since clients can converge on the same result with fewer instructions.

Backend compaction runs regularly on all synced clusters as part of Realm Sync.

Advanced backend compaction uses client sync states to further optimize a changeset. However, this optimization can prevent convergence for clients that have only seen a subset of history. As a result, advanced backend compaction only occurs for history older than the client maximum offline time.

Example

Consider the following realm history:

Original History
CREATE table2.object2
CREATE table1.object1
DELETE table2.object2

Because there is a gap in time between the CREATE and DELETE operations for table2.object2, a client could have synchronized the CREATE, but not the DELETE. As a result, compaction can remove the CREATE, but not the DELETE:

Compacted History
CREATE table1.object1
DELETE table2.object2

This guarantees client convergence. If the history didn't include the DELETE, clients that synced the CREATE would never delete table2.object2, but other clients would never create that object in the first place. Instead, all clients get a command to delete that object, and any clients who never created table2.object2 in the first place ignore the DELETE.

The following history uses advanced backend compaction to converge on the same state with even fewer steps:

Advanced Compacted History
CREATE table1.obj1

Because table2.object2 was deleted, it has no impact on the end state of the realm. As a result, advanced backend compaction can completely remove both the CREATE and the DELETE operations for table2.object2 from history. However, any clients that synchronized the CREATE operation but not the DELETE operation will not be able to converge to a common end state.

Advanced backend compaction only runs on Realm apps that specify a client maximum offline time. Only history older than the client maximum offline time experiences advanced backend compaction.

The client maximum offline time controls the age limit of history optimized with advanced backend compaction. This indirectly changes how long a client can remain offline between synchronization sessions with the backend. Clients that do not synchronize for more than the specified number of days may experience a client reset the next time they connect with the backend.

The lower the client maximum offline time, the sooner compaction can apply advanced backend compaction to history. The resulting optimizations usually result in less storage usage in the synced Atlas cluster.

Warning
Client Maximum Offline Time Causes Permanent Changes to History

Client maximum offline time enables advanced backend compaction for older history. This kind of compaction permanently changes affected history, and can cause client resets in the future even after disabling the feature. Use caution before enabling.

Sync should always converge at the same end state on all clients. In order to converge during a sync, clients require the full history of changes beginning immediately after their last sync. When a client does not sync for a long period of time, advanced backend compaction can alter the history in ways that prevent the client from converging. Since synchronization relies on all clients converging on a common result, such a client cannot synchronize.

Example

Consider the following example that contains a CREATE and a DELETE for table2.object2:

Original History
CREATE table2.object2
CREATE table1.object1
DELETE table2.object2

When this history becomes older than the client maximum offline time, advanced backend compaction optimizes the backend history into the following:

Advanced Compacted History
CREATE table1.obj1

However, a client contains this subset of history, containing only the CREATE operation for table2.object2:

Partial Client History
CREATE table2.object2
CREATE table1.object1

Since compaction pruned the DELETE operation for table2.object2, the client will never delete that object. The client cannot converge to the same state as the backend and all other clients.

As a result, the client must complete a client reset before it can resume synchronization. In a client reset scenario, the client deletes the client-local copy of a realm and downloads the current state of that realm from the backend. Synchronization then resumes using the new copy of the realm.

The client maximum offline time controls how long your backend waits before applying advanced backend compaction. After the specified number of days without syncing, clients may experience a client reset the next time they connect with the backend.

Applications that do not specify the client maximum offline time never apply advanced backend compaction. This means that clients can connect after any period of time offline -- weeks, months, or even years -- and synchronize changes. As time passes, frequently-edited realms accumulate many changes. With a large changeset, synchronization requires more time and data usage.

Advanced backend compaction causes permanent, irreversible changes to history. As a result, increasing the client maximum offline time does not immediately change the length of time before clients experience a client reset. Existing history has already been changed by advanced compaction, requiring a client reset. New history needs time to accumulate up to the new client maximum offline time.

Disabling the client maximum offline time feature stops additional advanced backend compaction, but history that has already been changed by advanced compaction will permanently cause client resets in clients.

Decreasing client maximum offline time also does not immediately change the length of time before clients experience a client reset. Client resets begin taking place earlier once the regularly scheduled compaction job applies advanced backend compaction to the newly eligible history.

  • Realm Sync uses space in your synced Atlas cluster to store change history.
  • Backend compaction reduces the space usage of this history.
  • Advanced backend compaction reduces the space usage of this history even more, but can cause client resets for clients who have not connected to the backend in more than the client maximum offline time (in days).
  • Apps that do not specify a client maximum offline time do not apply advanced backend compaction, so clients of any age can sync without experiencing a client reset.
Give Feedback
MongoDB logo
© 2021 MongoDB, Inc.

About

  • Careers
  • Legal Notices
  • Privacy Notices
  • Security Information
  • Trust Center
© 2021 MongoDB, Inc.