Navigation

Realm Sync Protocol

Overview

MongoDB Realm uses a protocol to correctly and efficiently sync data changes in real time across multiple clients that each maintain their own local Realm files. The protocol defines a set of pre-defined request types as well as a process by which a client, like a Realm SDK, can connect to a Realm application server and sync data.

Note

The Realm SDKs internally implement and manage the sync protocol, so for most applications you don’t need to understand the sync protocol to use Realm. This page covers the protocol at a high level and is not an implementation spec.

Key Concepts

Changeset

A changeset is a list of instructions that describe granular modifications made to a known object state or version by one or more write operations. Changesets are the base unit of the sync protocol. Synced realm clients send changesets to the Realm server whenever they perform a write operation. The server sends each connected client the changesets for write operations executed by other clients.

The Realm sync server accepts changesets from any connected sync client (including changes in a synced MongoDB cluster) at any time and uses an operational transformation algorithm to serialize changes into a linear order and resolve conflicting changesets before sending them to connected clients.

Operational Transformation

An operational transformation is a function that, given two changesets, produces a third changeset that represents logically applying one of the given changesets after the other. Realm uses operational transformation to resolve conflicts between changesets from different sync clients that apply to the same base state.

Realm is an offline-first local database even when sync is enabled, which means that any device may perform offline writes and upload the corresponding changesets later when network connectivity is re-established. The operational transformation algorithm is designed to gracefully handle changesets that arrive “out of order” with respect to the logical server clock such that every synced Realm file converges to the same version of each changed object.

Tip

An operational transformation on Realm changesets is analogous to a rebase operation in Git.

Client File Identifier

A client file identifier is a value that uniquely identifies a synced client Realm file and its corresponding server file. The server generates a client file identifier whenever an SDK requests one during its initial sync of a Realm file. Each identifier is a 64-bit, non-zero, positive signed integer strictly less than 2^63.

Note

The server guarantees that all identifiers generated on behalf of a particular server file are unique with respect to each other. The server is free to generate identical identifiers for two client files if they are associated with different server files.

Sync Session Process

To initiate, execute, and terminate a Realm Database sync session, a Realm SDK and application server send and receive a set of protocol-specific requests.

The SDK negotiates a WebSocket connection over HTTP and then establishes a sync session by sending BIND and IDENT requests to the server over the WebSocket connections. Once the session is established, the SDK and server send synced changesets for a given Realm file to each other via UPLOAD and DOWNLOAD messages. To end the session, the SDK sends an UNBIND request.

Realm SDK                      Realm App Server
    |                                  |
    |  <---- 1. HTTP Handshake ----->  |
    |                                  |
    |  --------- 2. BIND ----------->  |
    |                                  |
    |  <-- 3. IDENT (first time) ----  |
    |                                  |
    |  --------- 4. IDENT ---------->  |
    |                                  |
    |  <---- 5. UPLOAD/DOWNLOAD ---->  |
    |                                  |
    |  --------- 6. UNBIND --------->  |
1

Realm SDK Connects to the App Server

The sync protocol is primarily handled over a WebSocket connection between the SDK and the server. To establish a connection, the SDK sends a handshake HTTP request that includes the following:

  • a protocol version
  • a WebSocket key
  • a valid access token for an authenticated Realm application user

The server sends an HTTP 101 Switching Protocols response that specifies a WebSocket connection for the SDK. The rest of the sync protocol occurs over this connection.

2

Realm SDK Initiates a Sync Session

To begin a sync session, a Realm SDK sends a BIND request to a MongoDB Realm application server. The request identifies a specific local Realm file to sync and includes a WebSocket connection key that the server will use to open a bidirectional connection to the SDK.

If the SDK is attempting to sync a particular Realm file for the first time, it does not yet possess a server-generated client identifier for the file. In this case, the BIND request also indicates that the Realm server should allocate one.

3

Realm Allocates a New Client File Identifier

If a BIND request indicates that the SDK needs a client file identifier, the Realm server generates a unique value for the specified Realm file and sends it to the SDK in an IDENT response. When the SDK receives the IDENT, it stores the new client identifier persistently in the local Realm file.

An SDK only needs to request a client file identifier the first time it syncs each Realm file. For subsequent sync sessions, the SDK can use the persisted identifier.

4

Realm SDK Sends a Client Identifier

Once an SDK has initiated a sync session with a BIND request, it must identify the local Realm file that it wants to sync. To do this, the SDK sends the application server an IDENT message that contains the client file identifier. If the SDK has previously synced the realm with the server, it can specify the most recently synced server version to optimize the sync process.

When it receives the IDENT message, the server establishes the session. The SDK and server can can now freely send upload and download sync changesets at any time.

5

Realm SDK Uploads & Downloads Sync Changesets

Once a sync session is established, the SDK and server can freely send and receive UPLOAD and DOWNLOAD messages to sync changes whenever they occur.

The SDK sends an UPLOAD message for every changeset it applies except for those that it received in a DOWNLOAD message from the server.

When the server receives an UPLOAD message, it applies operational transformations to resolve any conflicts with other changesets and then applies the transformed changeset to the server version of the realm. This triggers the server to send DOWNLOAD messages to other connected clients, including the synced Atlas cluster which mirrors the server realm. A DOWNLOAD message groups one or more transformed changesets in chronological order from oldest to most recent according to the server’s history. The SDK applies the changesets in the same order.

6

Realm SDK Terminates the Sync Session

Once a sync session is established, Realm servers will continue to accept UPLOAD messages and send DOWNLOAD messages until the SDK terminates the session. To terminate a sync session, an SDK sends an UNBIND request to the Realm server.

Request Types

Client -> Server Messages

The following table describes the request types that a sync client can send to a Realm server:

Request Description
BIND

Starts a new sync session on the server and provides a signed authorization token for the current application user. If the client does not yet possess a client file identifier for the Realm file it wants to sync, this also indicates that the server should generate one and send it back to the client.

A client must send a BIND before it can send any other requests.

IDENT

Provides the client file identifier that indicates the following:

  • the Realm file to sync
  • the client realm’s current version
  • the client realm’s most recently synced server version

This request is related to but distinct from the IDENT message sent by the server when a client requests a client file identifier.

UPLOAD
Specifies one or more changesets for operations that occurred on the client. The changesets are listed by client version in increasing order.
TRANSACT
Specifies a changeset that describes a serialized transaction that occurred on the client. The client may not upload any other changesets until the server confirms or rejects the transaction.
UNBIND

Ends a running sync session.

A client may not send any other requests for an UNBOUND session.

MARK
Requests that the server notify the client when it has synced the latest changeset in the server history (at the time of the request).
REFRESH
Re-authorizes a current sync session with a new user token.
STATE_REQUEST
Requests that the server send one or more STATE messages, which the client uses to download the current server version of the Realm file. Clients issue state requests when they asynchronously open a synced realm.
CLIENT_VERSION_REQUEST
Requests that the server send the client version of the latest changeset that was sent by the client and processed by the server. This is most commonly used when an SDK executes a client reset.
PING

Indicates that the client is still connected and that the server should maintain the sync session. A client must send at least one PING to the server every 10 minutes. The server acknowledges to each PING with a PONG.

If the server has not received a PING from a client in more than 10 minutes, it considers the client to be disconnected and may automatically end the session.

Server -> Client Messages

The following table describes the request types that the Realm server can send to a sync client:

Request Description
IDENT
Provides a client file identifier that the server generated in response to a BIND that requested the identifier.
DOWNLOAD

Specifies one or more changesets (up to 16MB total) for operations that occurred on other clients. The changesets are listed by server version in increasing order.

The changesets in a DOWNLOAD may not be the exact changesets uploaded by other clients. Instead, they may be equivalent changesets output by Realm’s operational transformation algorithm.

UNBOUND
Specifies that the server ended a sync session in response to an UNBIND.
TRANSACT
Indicates whether or not the server successfully processed a changeset specified in a TRANSACT from the client.
MARK
Indicates that the server has sent the client the latest changeset that was in the server history when the server received a MARK from the client.
STATE
Contains one or more segments of encoded data that the client can concatenate to construct the latest server version of the realm. Sent in response to a STATE_REQUEST.
CLIENT_VERSION
Specifies the client version of the latest changeset that was sent by the client and processed by the server. Sent in response to a CLIENT_VERSION_REQUEST.
ERROR
Indicates that the server encountered an issue that appears to have been caused by the connected client.
PONG
Acknowledges a PING. If a client does not receive a PONG acknowledgement, it indicates that the client cannot currently communicate with the server over the network and that the server may not have received the corresponding PING.