Manage Namespace Metadata Catalog for Wildcard Collections


The Atlas Data Lake is available as a Beta feature. The product and the corresponding documentation may change at any time during the Beta stage. For support, see Atlas Support.

The Atlas Data Lake namespace catalog contains namespace metadata (such as databases, collections, and partitions) for each wildcard (*) collection in each data store in the Data Lake storage configuration. For commands and queries on wildcard collections, Atlas Data Lake returns the data from the catalog or returns an error if the data store does not exist.

When objects that are included in a wildcard collection specification are modified in your S3 bucket, the catalog must be updated for you to see the changes when you run list commands against wildcard collections. By default, the catalog is refreshed every hour for all active Data Lakes. In addition, the namespace metadata catalog is refreshed in the background when you:

You can determine when the catalog was last updated by using the catalogInfo command and can manually update the catalog by using the updateCatalog command. You must have the atlasAdmin role to update the catalog.