Navigation

Pipelines

MongoDB Stitch provides the ability to define a sequence of actions to perform as a pipeline.

  • Pipelines are written in simple JSON.
  • Pipelines can be constructed in the UI and referenced by name in code, or coded directly with SDKs.
  • Pipelines can use expansions to pass information from one stage to another or incorporate information from outside the pipeline stage
    • Information can be passed from one stage to another with let.
    • Information about requesting user can be incorporated with %%user.
    • Global variables can be defined in Stitch and referenced with %%values.
    • Information about the current or prior state of a document or field can be incorporated through MongoDB expansions.
    • Some services like Twilio can execute a pipeline as a response to an incoming webhook.

Pipeline Stages

A pipeline consists of stages. Pipeline stages are executed in order. Each stage specifies an action to perform via a service. Each stage passes its output on as input for the next stage, and the final result, if any, is returned to the application client. A pipeline can consist of stages that use different MongoDB Stitch services.

Syntactically, a pipeline consists of an array of documents, one document per stage. Each document specifies a service, an action, and any required arguments for the action. Optionally, you can also define variables to use in the arguments.

[ { "service": <servicename>, "action": <action>, "args": <argument document>, "let": <variable document> }, ... }]

Tip

In the stages, the action arguments can make use of variables (%%vars) defined for the stage as well as defined constants and expansions.

Service Actions

Note

For details on a specific action, including syntax and examples, click on the specific operator to go to its reference page.

Built In Actions

Action Description
binary Decodes base64 or hexadecimal encoded data and outputs as binary data stream.
encode Encodes incoming binary data into specified format.
expr Evaluates an expression against each input document and outputs the evaluated documents.
literal Explicitly defines its output documents.
match Filters its input documents.
namedPipeline Executes a named pipeline, given its name and arguments.
null Does nothing and outputs nothing.
project Determines which fields to include or exclude in the output documents.

MongoDB Service Actions

Action Description
aggregate Executes an aggregation pipeline.
delete Deletes documents from a MongoDB collection.
find Queries a collection in MongoDB.
insert Inserts the stage’s input documents into the specified MongoDB collection.
update Modifies one or more documents in a MongoDB collection.

Amazon S3 Service Actions

Action Description
put Uploads a binary data to an S3 bucket.
signPolicy Generates a document that contains the policy, signature, and accessKeyId, which you can use to issue a POST request directly to S3.

Amazon SES Service Actions

Action Description
send Sends an email message to a specified address.

HTTP Service Actions

Action Description
delete Corresponds to the HTTP method DELETE.
get Corresponds to the HTTP method GET.
head Corresponds to the HTTP method HEAD.
patch Corresponds to the HTTP method PATCH.
post Corresponds to the HTTP method POST.
put Corresponds to the HTTP method PUT.

Mailgun Service Actions

Action Description
send Sends an email message to a specified address.

Slack Service Actions

Action Description
post Sends a Slack message to a specified channel.

Twilio Service Actions

Action Description
send Sends an email message to a specified address.

Clients

You can call the pipeline from your application client to execute the pipeline.

JavaScript SDK

To execute a pipeline, use the StitchClient.executePipeline() method. For example:

stitchClient.executePipeline([
     {
       "service":"","action":"literal",
       "args":{ "items":[
          {"customer":"A","item": "oranges","qty":10},
          {"customer":"B","item":"grapes", "qty": 50},
          {"customer":"A","item": "oranges","qty":10},
          {"customer":"B","item":"apples", "qty": 10}
       ] }
     },
     { "service":"mongodb-atlas","action":"insert","args":{"database":"my_db","collection":"orders"} }
   ]);

Android SDK

To execute a pipeline, use the StitchClient.executePipeline() method.For example:

final List<PipelineStage> pipeline = new ArrayList<>();

final Map<String, Object> binaryStage = new HashMap<>();
binaryStage.put("encoding", "base64");
binaryStage.put("data", Base64.encodeToString(text.getBytes(), Base64.DEFAULT));
pipeline.add(new PipelineStage("binary", binaryStage));

stitchClient.executePipeline(pipeline).addOnCompleteListener( //... );

The Debug Console

The MongoDB Stitch console provides a facility for testing pipelines in the Debug Console. You can reach the Debug Console through the admin console’s left-side navigation.

To use the Debug Console for testing pipelines, select an application user for the pipeline to run as at the top of the screen, then assemble your pipeline stages with the form inputs.

Examples

Built-In Actions

A very simple pipeline is one with a single stage in which the built-in literal action returns an array of items.

[
  {
    "action": "literal",
    "args": {
      "items": [
        { "type": "apples", "qty": 25 },
        { "type": "oranges", "qty": 50 }
      ]
    }
  }
]

The above pipeline returns the following to the client:

[
  {
    "type": "apples",
    "qty": {
      "$numberDouble": "25"
    }
  },
  {
    "type": "oranges",
    "qty": {
      "$numberDouble": "50"
    }
  }
]

Note

When using the literal action, the args object must include an array called items.

The MongoDB Service

In order to use the MongoDB service in a pipeline stage, you must have rules set up for each namespace included in the operation.

The following example uses a collection called cars in a database called vehicles, with the following documents:

{ "_id": 1, "make": "Honda", "model": "Accord", "doors": 4 }
{ "_id": 2, "make": "Ford", "model": "Mustang", "doors": 2 }
{ "_id": 3, "make": "Toyota", "model": "Camry", "doors": 4 }

The following pipeline uses the service mdb1 to find 2-door cars, then uses the built-in project action to display only the make and model fields.

[
  {
    "service": "mdb1"
    "action": "find",
    "args": {
      "collection": "cars",
      "database": "vehicles",
      "query": {
        "doors": 2
      }
    },
  },
  {
    "action": "project",
    "args": {
      "projection": {
        "make": 1,
        "model": 1
      }
    }
  }
]

The above pipeline returns the following array to the client:

[
  {
    "make": "Ford",
    "model": "Mustang"
  }
]

Binary Stages

One use case for a MongoDB Stitch application could be to allow users to upload binary data to an Amazon S3 bucket. You can accomplish this in a pipeline which uses the S3 service and the built-in binary action.

The following example takes a hex-encoded string and decodes it to binary. The binary data passes to the next stage, which uses a put action to upload it to an S3 bucket.

[
  {
    "action": "binary",
    "args": {
      "data": "68656c6c6f",
      "encoding": "hex"
    }
  },
  {
    "service": "my-s3-service",
    "action": "put",
    "args": {
      "contentType": "text/plain",
      "acl": "public-read",
      "bucket": "my-bucket-name",
      "key": "test-object"
    }
  }
]

The document returned to the client has the following form:

{ "location": "https://my-bucket-name.s3-us-west-2.amazonaws.com/test-object" }

Note

In the above example, the key argument to the S3 service represents the path to the object in the S3 bucket, relative to server root. It must not begin with /.

To take binary data and convert it to an encoded string, use the built-in encode action.

The following pipeline stage accepts incoming binary data until EOF is reached and returns it as a single document.

{
  "action": "encode",
  "args": {
    "encoding": "base64"
  }
}

The output document has the following form:

{ data: "<encoded data>"}

The Twilio Service

The following example uses a MongoDB database called people and a collection called users with the following documents:

{ "_id" : 1, "name" : "Amos", "city" : "Boston", "phone" : "+15558907614" }
{ "_id" : 2, "name" : "Barbara", "city" : "Atlanta", "phone" : "+15558340186" }
{ "_id" : 3, "name" : "Chet", "city" : "Boston", "phone" : "+15553528799" }

The following pipeline combines a find call to a MongoDB service called mongodb-atlas with a send call to a Twilio service called tw1. It sends a text message to all the users who meet the specified criteria.

The first stage retrieves all documents with Boston in the city field and passes them on to the next stage.

The second stage uses a let statement to iterate over all the documents passed to it from the first stage. The %%item expansion holds the data for each document as it passes through, and the %%vars expansion makes the %%item data available to the stage arguments.

[
  {
    "service": "mongodb-atlas"
    "action": "find",
    "args":
      {
        "database": "people",
        "collection": "users",
        "query":
          {
            "city": "Boston"
          }
      }
  },
  {
    "service": "tw1",
    "action": "send",
    "let":
      {
        "toPhone": "%%item.phone"
      },
    "args":
      {
        "to": "%%vars.toPhone",
        "from": "+15553776250",
        "body": "Looks like rain today. Bring an umbrella!"
      }
  }
]

The two users from Boston receive a text message via Twilio. A successful pipeline response returns an empty array to the client.