Navigation

Partition Atlas Data into Realms

Overview

Your Realm app’s linked MongoDB Atlas cluster contains all of the data stored by your application: your application data, client data, custom user data, administrative data, and more.

Client applications only need the data relevant to the current user. Most users only have permission to access a limited subset of the data in a linked cluster. Some data, like user settings and personal information, are only relevant to a single user.

Partitioning helps you turn the large amount of data stored by your application into the small amount of data needed by individual clients, called realms.

../../_images/partitioning.svg

Partition shape data in an MongoDB Atlas cluster using a color partition key with partition values red, blue, and green.

Partition Keys

MongoDB Realm uses partition keys to map MongoDB documents in a synced MongoDB Atlas cluster into Realm objects in realms. MongoDB Realm uses a special field called a partition key to determine the realm membership of MongoDB documents. Your partition key should exist in every document that your application needs to synchronize between clients and Atlas. The partition key field name is universal across a Realm app – every synced document uses the same field name to partition.

Choose a Partition Key

Your partition key depends entirely on your use case: some applications only require a per-user permission scheme, but many require permissions based on collections of users. Generally, a few guidelines exist to choose partition keys:

  • Realm clients should never modify the partition value directly. Any field that clients can modify cannot be used as a partition key.
  • Changing a partition value is an expensive operation, requiring a deletion of the object from one realm and an insertion of the same object into another realm. As a result, you should choose a partition key that rarely, if ever, needs to change.
  • Partition keys use the same field name in every synced document. Pick a field name that is unlikely to collide with an another field name in any Realm object’s model. You can begin the field name with an underscore, (e.g. _partitionKey) to decrease the likelihood of collisions and indicate the importance of the field in the Object Model itself.
  • Partition keys must have a value of one of the supported types for partition keys.

For more information about how to choose a partition key for your application, see Partition Strategies.

Define a Partition Key

You can define a partition key when enabling Realm Sync for your Realm app. After selecting a cluster to sync, choose your partition key. You can either use an existing field or create a new field.

The Realm Schema for each synced collection of your synced cluster should include the partition key for your application. However, you don’t have to include the partition key in all collections; Realm Sync ignores any collections that lack a Realm Schema as well as any documents that lack a valid value for the partition key.

Partition Values

Each unique partition value, or value of a partition key, maps to a different realm. Documents that share the same partition value belong to the same realm. All documents in a realm share the same read and write permissions.

Map Documents to Realms

Since all documents that share the same partition value also share the same permissions for each user, you should select a key whose unique values correspond with permissions within your application. Consider which documents each user needs to read, and which documents each user needs to write. What separates one user’s data from another? The concepts of ownership (which users can change which data?) and access (which users can see which data?) decide how you should partition your data.

Partition Strategies

Most applications access data in similar ways. A few common patterns, applied individually or in combination, cover a vast majority of permissioning use cases:

  • User Realms: If your application stores data privately for each individual user, you could create an owner_id field containing a specific user ID for every document. Selecting the owner_id field as your partition key would partition your database into realms corresponding to individual users.
  • Team Realms: If your application shares resources across teams, you could create a teams collection that maps user IDs to team IDs. You could then create a team_id field containing a specific team ID for every document you want to sync. Selecting the team_id field as your partition key would partition your database into realms containing data for entire teams.
  • Public Realms: It’s often helpful to store read-only data viewable to all users; for this kind of use case, you can define a special partition value in your permissions, such as PUBLIC, that all users sync with read permissions only.

You can combine these partitioning strategies to create applications that make some data publicly available, like announcements or an organizational chart, but restrict other data to privileged users.

To combine strategies, you can use a generic partition key field name like _partitionKey with a partition value structured like a query string. For example:

_partitionKey: "team_id=1234"
_partitionKey: "user_id=abcdefg"

Realm Rules expressions can check the prefix before looking up the team or user ID in the appropriate lookup collection to determine whether or not a given user has access to a realm. Structuring your partition values like a query string lets you take advantage of multiple partitioning strategies at once, providing the power of user, team, and public realms all at once. See the partition key examples for more information about partitioning strategies.

Partitioning Limitations

Partition keys and values are subject to certain limitations:

  • Partition keys must be one of the following types:

    • String
    • ObjectID
    • Int
    • Long
  • Once you have chosen a partition key and enabled sync, you cannot reassign the partition key to a different field. To use a different field as your partition key:

    1. Disable Realm Sync.
    2. Enable Realm Sync again with the new field selected as your partition key.

    Warning

    Disabling and re-enabling Sync will trigger a client reset on all clients.

  • Once you have chosen a partition key and enabled Sync, you cannot change the name of the field you selected as your partition key. To alter the field name of your partition key:

    1. Disable Realm Sync.
    2. Change the field name.
    3. Enable Realm Sync again with the updated field name selected as your partition key.

    Warning

    Disabling and re-enabling Realm Sync will trigger a client reset on all clients.

  • If you change a partition value in Atlas, MongoDB Realm interprets the change as a delete, then an insert. First, MongoDB Realm deletes the corresponding Realm object from the object’s original realm. Next, MongoDB Realm inserts a new Realm object containing the same data into the object’s new realm.

    Warning

    Changing a partition value in Atlas will lose any un-synced client updates to the object.

  • Avoid changing a partition value in a Realm object using the Realm SDK because it triggers a client reset.

  • Documents linked to each other using a relationship produce “dangling links” with a null value if the documents do not belong to the same partition. If your application creates relations that span multiple realms, you should be prepared to handle null values for all such relations.

    Warning

    Relational object fields that lack a corresponding object, known as “dangling links,” resolve to null values in the Realm SDKs. Dangling links can occur if a relation spans multiple realms These relations are valid in MongoDB Atlas which contains both objects in the relation, but not in any single client realm. To prevent dangling links, make sure that each object in a relation uses the same partition value.

Partitioning Examples

Because realms correspond to collections of documents that share the same permissions, partitioning can vary between Realm apps.

Example: Partitioning by User

Consider the following documents in three MongoDB collections describing user data in a music app:

collection playlists: [
   { name: "Work", owner_id: "dog_enthusiast_95", song_ids: [ ... ] }
   { name: "Party", owner_id: "cat_enthusiast_92", song_ids: [ ... ] }
   { name: "Dance", owner_id: "troll_2", song_ids: [ ... ] }
   { name: "Soup Tunes", owner_id: "dog_enthusiast_95", song_ids: [ ... ] }
   { name: "Disco Anthems", owner_id: "PUBLIC", song_ids: [ ... ] }
   { name: "Deep Focus", owner_id: "PUBLIC", song_ids: [ ... ] }
]

collection follows: [
   { owner_id: "dog_enthusiast_95", playlist_ids: [ ... ] }
   { owner_id: "cat_enthusiast_92", playlist_ids: [ ... ] }
   { owner_id: "troll_2", playlist_ids: [ ... ] }
   { owner_id: "OP", playlist_ids: [ ... ] }
]

collection dispositions: [
   { owner_id: "dog_enthusiast_95", song_id: 3, rating: -1 }
   { owner_id: "cat_enthusiast_92", song_id: 1, rating: 1 }
   { owner_id: "troll_2", song_id: 2, rating: 1 }
   { owner_id: "dog_enthusiast_95", song_id: 1, rating: 1 }
]

In this application, the owner_id field is a good candidate for a partition key. This field works well as a partition key because it splits documents into realms that make sense given user permissions and it makes it easy for users to sync a limited number of realms:

  • Every user has read and write access to the realm containing objects owned by them – playlists they’ve made and song ratings.
  • Every user has read access to the realm for partition value PUBLIC, which contains all playlists that are available to all users.
  • Users sync to two realms: the realm for the partition value corresponding to their user ID, and the realm for the PUBLIC partition value. This minimizes data synced over the network to only the data relevant for each Realm user.

Partitioning this data by owner_id results in the following realms:

realm dog_enthusiast_95: [
   { owner_id: "dog_enthusiast_95", playlist_ids: [ ... ] }
   { name: "Work", owner_id: "dog_enthusiast_95", song_ids: [ ... ] }
   { name: "Soup Tunes", owner_id: "dog_enthusiast_95", song_ids: [ ... ] }
   { owner_id: "dog_enthusiast_95", song_id: 3, rating: -1 }
   { owner_id: "dog_enthusiast_95", song_id: 1, rating: 1 }
]

realm cat_enthusiast_92: [
   { owner_id: "cat_enthusiast_92", playlist_ids: [ ... ] }
   { name: "Party", owner_id: "cat_enthusiast_92", song_ids: [ ... ] }
   { owner_id: "cat_enthusiast_92", song_id: 1, rating: 1 }
]

realm troll_2: [
   { owner_id: "troll_2", playlist_ids: [ ... ] }
   { owner_id: "troll_2", song_id: 2, rating: 1 }
   { name: "Dance", owner_id: "troll_2", song_ids: [ ... ] }
]

realm OP: [
   { owner_id: "OP", playlist_ids: [ ... ] }
]

realm PUBLIC: [
   { name: "Disco Anthems", owner_id: "PUBLIC", song_ids: [ ... ] }
   { name: "Deep Focus", owner_id: "PUBLIC", song_ids: [ ... ] }
]

Example: Partitioning Data Across Groups

Consider the following documents in MongoDB collections describing inventory and sales at a paper company:

collection branches: [
   { name: "Scranton", partitionKey: "PUBLIC", salespeople: [ "Jim", "Pam", "Dwight" ] }
   { name: "Albany", partitionKey: "PUBLIC", salespeople: [ "Karen" ] }
   { name: "Utica", partitionKey: "PUBLIC", salespeople: [ "Jeff" ] }
   { name: "Yonkers", partitionKey: "PUBLIC", salespeople: [ "Arnold" ] }
]

collection inventory: [
   { type: "A4", partitionKey: "branch=Scranton", volume: 326}
   { type: "A4", partitionKey: "branch=Albany", volume: 567}
   { type: "A4", partitionKey: "branch=Utica", volume: 48}
]

collection sales: [
   { item: "A4", partitionKey: "salesperson=Jim", volume: 33, customer: "school" }
   { item: "A4", partitionKey: "salesperson=Karen", volume: 78, customer: "law firm" }
   { item: "A4", partitionKey: "salesperson=Jim", volume: 57, customer: "gym" }
   { item: "A4", partitionKey: "salesperson=Jeff", volume: 15, customer: "mom" }
   { item: "A4", partitionKey: "salesperson=Arnold", volume: 54, customer: "dad" }
]

collection delivery: [
   { item: "A4", partitionKey: "delivery_truck=3", volume: 33, customer: "school" }
   { item: "A4", partitionKey: "delivery_truck=3", volume: 57, customer: "gym" }
   { item: "A4", partitionKey: "delivery_truck=7", volume: 12, customer: "law firm" }
]

collection catalog: [
   { item: "A4", partitionKey: "PUBLIC", price: .25 }
   { item: "A5", partitionKey: "PUBLIC", price: .75 }
   { item: "legal", partitionKey: "PUBLIC", price: .35 }
   { item: "panama", partitionKey: "PRIVATE", price: 1.05 }
]

collection leads: [
   { customer: "yellow pages", contact: "123-456-7890", partitionKey: "branch=Scranton" }
   { customer: "newspaper", contact: "456-456-7890", partitionKey: "branch=Scranton" }
   { customer: "diner", contact: "345-623-5467", partitionKey: "branch=Albany" }
   { customer: "mom", contact: "342-576-6733", partitionKey: "salesperson=Jeff" }
   { customer: "NYSDEC", contact: "145-674-7890", partitionKey: "branch=Utica" }
]

For this data, there is no single collection of owners – depending on the collection, data might be owned by:

  • a branch of the company, such as Scranton
  • a salesperson, such as Jim
  • a delivery truck, such as truck number 7
  • the company in general, public or private

To partition this data into realms, you can create partition keys containing prefixes that indicate which kind of entity owns the data. In this case, the application uses the categories of branch, salesperson, delivery truck, or public/private. You can also create a unique ID for each entity across categories and map those IDs in a separate entity metadata collection.

It makes sense to partition data in this way because it separates data into groups of documents with the same permissions. Additionally, clients only sync a small number of relevant realms for each use case:

  • A mobile client assigned to a delivery truck has read and write access to the inventory of their delivery truck, and syncs the realms correponding to their delivery truck’s ID. No other information is relevant for that delivery truck, so they only access the information they need.
  • Every salesperson has read and write access to their own sales data and syncs the realm corresponding to their own sales data.
  • Every branch warehouse can read and write data in their own inventory realm.
  • Every user can read public company information located in the PUBLIC realm, and only company administrators can read and write public or private company information.
  • Because most users only read and write to one or two realms, this partitioning scheme only exposes data to those who need it. The partitioning remains flexible enough to accommodate future features.

Partitioning this data by partitionKey results in the following realms:

realm PUBLIC: [
   { name: "Scranton", partitionKey: "PUBLIC", salespeople: [ "Jim", "Pam", "Dwight" ] }
   { name: "Albany", partitionKey: "PUBLIC", salespeople: [ "Karen" ] }
   { name: "Utica", partitionKey: "PUBLIC", salespeople: [ "Jeff" ] }
   { name: "Yonkers", partitionKey: "PUBLIC", salespeople: [ "Arnold" ] }
   { item: "A4", partitionKey: "PUBLIC", price: .25 }
   { item: "A5", partitionKey: "PUBLIC", price: .75 }
   { item: "legal", partitionKey: "PUBLIC", price: .35 }
]

realm PRIVATE: [
   { item: "panama", partitionKey: "PRIVATE", price: 1.05 }
]

realm branch=Scranton: [
   { type: "A4", partitionKey: "branch=Scranton", volume: 326}
   { customer: "yellow pages", contact: "123-456-7890", partitionKey: "branch=Scranton" }
   { customer: "newspaper", contact: "456-456-7890", partitionKey: "branch=Scranton" }
]

realm branch=Albany: [
   { type: "A4", partitionKey: "branch=Albany", volume: 567}
   { customer: "diner", contact: "345-623-5467", partitionKey: "branch=Albany" }
]

realm branch=Utica: [
   { type: "A4", partitionKey: "branch=Utica", volume: 48}
   { customer: "NYSDEC", contact: "145-674-7890", partitionKey: "branch=Utica" }
]

realm salesperson=Jim: [
   { item: "A4", partitionKey: "salesperson=Jim", volume: 33, customer: "school" }
   { item: "A4", partitionKey: "salesperson=Jim", volume: 57, customer: "gym" }
]

realm salesperson=Karen: [
   { item: "A4", partitionKey: "salesperson=Karen", volume: 78, customer: "law firm" }
]

realm salesperson=Jeff: [
   { item: "A4", partitionKey: "salesperson=Jeff", volume: 15, customer: "mom" }
   { customer: "mom", contact: "342-576-6733", partitionKey: "salesperson=Jeff" }
]

realm salesperson=Arnold: [
   { item: "A4", partitionKey: "salesperson=Arnold", volume: 54, customer: "dad" }
]

realm delivery_truck=3: [
   { item: "A4", partitionKey: "delivery_truck=3", volume: 33, customer: "school" }
   { item: "A4", partitionKey: "delivery_truck=3", volume: 57, customer: "gym" }
]

realm delivery_truck=7: [
   { item: "A4", partitionKey: "delivery_truck=7", volume: 12, customer: "law firm" }
]