Supported Partition Attribute Types¶
The following table lists the supported data types for partition attributes and
an example databases.[n].collections.[n].dataSources.[n].path
for each data type:
note
When specifying the
databases.[n].collections.[n].dataSources.[n].path, use the
delimiter specified in stores.[n].delimiter.
| Key | Data Type | Example |
|---|---|---|
string | Parses the filename as a string. This is the default data type. If a data type is not specified for a partition attribute, Data Lake interprets the partition attribute as a string. | filename: path: OR path: In the above |
int | Parses the filename as an integer. | filename: path: In the above example, |
isodate | Parses the filename in RFC 3339 format as an ISO-8601 format date. | filename: path: In the above example, |
epoch_secs | Parses the filename as a Unix timestamp in seconds. | filename: path: In the above example, |
epoch_millis | Parses the filename as a Unix timestamp in milliseconds. | filename: path: In the above example, |
objectid | Parses the filename as an ObjectId. | filename: path: In the above example, |
uuid | Parses the filename as a UUID of binary subtype 4. | filename: path: In the above example, |
note
Atlas Data Lake supports the Package Syntax for regular expressions in the path to the filename.
Parsing Null Values from Filenames¶
If there is an empty string ("") in the place of an attribute in the path
to the file, Data Lake automatically parses it as a null value for all the
attribute types except string. For example, consider the following S3
data store:
/records/january/1.json
/records/february/1.json
/records//1.jsonFor the path /records/{month string}/*, Data Lake does not add any
computed fields for the month attribute to documents generated
from the third record in the above store.
Parsing Padded Numbers from Filenames¶
For attribute types like int, epoch_millis, and epoch_secs,
if you want Data Lake to correctly parse numeric values that are padded
with leading zeros in the path to the file, specify the number
of digits in the padded value using regular expressions. For example,
consider a S3 store with the following files:
|--users
|--001.json
|--002.json
...The following path syntax uses a regular expression with the
int attribute type to specify the number of digits in the
filename:
/users/{user_id int:\\d{3}}