Realm Sync Protocol¶
MongoDB Realm uses a protocol to correctly and efficiently sync data changes in real time across multiple clients that each maintain their own local Realm files. The protocol defines a set of pre-defined request types as well as a process by which a client, like a Realm SDK, can connect to a Realm application server and sync data.
The Realm SDKs internally implement and manage the sync protocol, so for most applications you don't need to understand the sync protocol to use Realm. This page covers the protocol at a high level and is not an implementation spec.
A changeset is a list of instructions that describe granular modifications made to a known object state or version by one or more write operations. Changesets are the base unit of the sync protocol. Synced realm clients send changesets to the Realm server whenever they perform a write operation. The server sends each connected client the changesets for write operations executed by other clients.
The Realm sync server accepts changesets from any connected sync client (including changes in a synced MongoDB cluster) at any time and uses an operational transformation algorithm to serialize changes into a linear order and resolve conflicting changesets before sending them to connected clients.
An operational transformation is a function that, given two changesets, produces a third changeset that represents logically applying one of the given changesets after the other. Realm uses operational transformation to resolve conflicts between changesets from different sync clients that apply to the same base state.
Realm is an offline-first local database even when sync is enabled, which means that any device may perform offline writes and upload the corresponding changesets later when network connectivity is re-established. The operational transformation algorithm is designed to gracefully handle changesets that arrive "out of order" with respect to the logical server clock such that every synced Realm file converges to the same version of each changed object.
An operational transformation on Realm changesets is analogous to a rebase operation in Git.
Client File Identifier¶
A client file identifier is a value that uniquely identifies a synced client Realm file and its corresponding server file. The server generates a client file identifier whenever an SDK requests one during its initial sync of a Realm file. Each identifier is a 64-bit, non-zero, positive signed integer strictly less than 2^63.
The server guarantees that all identifiers generated on behalf of a particular server file are unique with respect to each other. The server is free to generate identical identifiers for two client files if they are associated with different server files.
Sync Session Process¶
To initiate, execute, and terminate a Realm Database sync session, a Realm SDK and application server send and receive a set of protocol-specific requests.
The SDK negotiates a WebSocket connection over HTTP and then establishes a sync
session by sending
requests to the server over the WebSocket connections. Once the session is
established, the SDK and server send synced changesets for a given
Realm file to each other via
DOWNLOAD messages. To end the session, the SDK sends an
Realm SDK Realm App Server | | | <---- 1. HTTP Handshake -----> | | | | --------- 2. BIND -----------> | | | | <-- 3. IDENT (first time) ---- | | | | --------- 4. IDENT ----------> | | | | <---- 5. UPLOAD/DOWNLOAD ----> | | | | --------- 6. UNBIND ---------> |
Realm SDK Connects to the App Server¶
The sync protocol is primarily handled over a WebSocket connection between the SDK and the server. To establish a connection, the SDK sends a handshake HTTP request that includes the following:
- a protocol version
- a WebSocket key
- a valid access token for an authenticated Realm application user
The server sends an HTTP 101 Switching Protocols response that specifies a WebSocket connection for the SDK. The rest of the sync protocol occurs over this connection.
Realm SDK Initiates a Sync Session¶
To begin a sync session, a Realm SDK sends a
request to a MongoDB Realm application server. The request identifies a
specific local Realm file to sync and includes a WebSocket connection key that
the server will use to open a bidirectional connection to the SDK.
If the SDK is attempting to sync a particular Realm file for the first time,
it does not yet possess a server-generated client identifier for the file. In
this case, the
BIND request also indicates that the
Realm server should allocate one.
Realm Allocates a New Client File Identifier¶
BIND request indicates that the SDK needs a client
file identifier, the Realm server generates a unique value for the specified
Realm file and sends it to the SDK in an
response. When the SDK receives the
IDENT, it stores
the new client identifier persistently in the local Realm file.
An SDK only needs to request a client file identifier the first time it syncs each Realm file. For subsequent sync sessions, the SDK can use the persisted identifier.
Realm SDK Sends a Client Identifier¶
Once an SDK has initiated a sync session with a
request, it must identify the local Realm file that it wants to sync. To do
this, the SDK sends the application server an
message that contains the client file identifier. If the SDK has previously
synced the realm with the server, it can specify the most recently synced
server version to optimize the sync process.
When it receives the
IDENT message, the server
establishes the session. The SDK and server can can now freely send upload and
download sync changesets at any time.
Realm SDK Uploads & Downloads Sync Changesets¶
When the server receives an
UPLOAD message, it applies
operational transformations to resolve
any conflicts with other changesets and then applies the transformed changeset
to the server version of the realm. This triggers the server to send
DOWNLOAD messages to other connected clients, including
the synced Atlas cluster which mirrors the server realm. A
DOWNLOAD message groups one or more transformed
changesets in chronological order from oldest to most recent according to the
server's history. The SDK applies the changesets in the same order.
Client -> Server Messages¶
The following table describes the request types that a sync client can send to a Realm server:
Starts a new sync session on the server and provides a signed authorization token for the current application user. If the client does not yet possess a client file identifier for the Realm file it wants to sync, this also indicates that the server should generate one and send it back to the client.
A client must send a BIND before it can send any other requests.
Provides the client file identifier that indicates the following:
This request is related to but distinct from the
Specifies one or more changesets for operations that occurred on the client. The changesets are listed by client version in increasing order.
Specifies a changeset that describes a serialized transaction that occurred on the client. The client may not upload any other changesets until the server confirms or rejects the transaction.
Ends a running sync session.
A client may not send any other requests for an
Requests that the server notify the client when it has synced the latest changeset in the server history (at the time of the request).
Re-authorizes a current sync session with a new user token.
Requests that the server send one or more
Requests that the server send the client version of the latest changeset that was sent by the client and processed by the server. This is most commonly used when an SDK executes a client reset.
Indicates that the client is still connected and that the server should
maintain the sync session. A client must send at least one PING to the
server every 10 minutes. The server acknowledges to each PING with a
If the server has not received a PING from a client in more than 10 minutes, it considers the client to be disconnected and may automatically end the session.
Server -> Client Messages¶
The following table describes the request types that the Realm server can send to a sync client:
Specifies one or more changesets (up to 16MB total) for operations that occurred on other clients. The changesets are listed by server version in increasing order.
The changesets in a DOWNLOAD may not be the exact changesets uploaded by other clients. Instead, they may be equivalent changesets output by Realm's operational transformation algorithm.
Specifies that the server ended a sync session in response to an
Indicates whether or not the server successfully processed a changeset specified in a
Contains one or more segments of encoded data that the client can concatenate to construct the latest server version of the realm. Sent in response to a
Specifies the client version of the latest changeset that was sent by the client and processed by the server. Sent in response to a
Indicates that the server encountered an issue that appears to have been caused by the connected client.