What is it?

Note

These docs assume readers have a basic knowledge of Python and are familiar with the Python package, xarray. If you’re not familiar with xarray, check out the excellent xarray tutorial webpage.

The short, high-level description

The ACCESS-NRI catalog is essentially a table of climate data products that exist on Gadi. Each entry in the table corresponds to a different product, and the columns contain attributes associated with each product–things like the models, frequencies and variables available. Users can search on the attributes to find the products that might be useful to them. For example, a user might want to know which data products contain variables X, Y and Z at monthly frequency. The ACCESS-NRI catalog enables users to find products that satisfy their query and to subsequently load their data without having to know the location and structure of the underlying files.

Hang on, I don’t have access to Gadi!

Never fear, you may still have a use for access-nri-intake-catalog! You can use the Datastore Builders in the package to create Intake-ESM datastores for your own experiments. See the Quickstart tutorial on building your own Intake-ESM datastores for details.

The more detailed description

The ACCESS-NRI catalog provides a catalog of Intake sources and associated metadata.

Wait, what are “Intake sources”?

“Intake” is a Python package that provides a general interface for loading data in Python. Intake provides a set of data loaders (called “drivers”) that allow users to load a wide range of data formats into familiar Python data structures (e.g. a pandas dataframe or xarray Dataset) using the exact same code. Some drivers are built into Intake, and some are provided by other “plugin” packages. Documentation of the Intake package can be found here and a list of available Intake drivers can be found here.

“Intake sources” describe data that can be loaded using an Intake driver. For example, an Intake source might describe a simple csv file to be opened using the inbuilt Intake “csv” driver, or it might describe a set of netcdf files to be opened using the plugin Intake-ESM “esm_datastore” driver. In fact, the ACCESS-NRI catalog itself is a type of Intake source that uses the plugin Intake-dataframe-catalog “df_catalog” driver.

The entries in the ACCESS-NRI catalog are Intake-ESM datastores for climate data products that exist on Gadi.

Wait, what are “Intake-ESM datastores”?

Intake-ESM datastores are a type of Intake source that describes a climate data product comprising many files (e.g. netcdf files). Users can search across metadata associated with each file (e.g. a variable name) and open contiguous files into xarray Datasets for subsequent analysis. For example, NCI provides Intake-ESM datastores for the CMIP5 and CMIP6 data collections available on Gadi. In their documentation they demonstrate how to execute a search across these datastores to find and open a few datasets.

Intake-ESM datastores are often also referred to as Intake-ESM “catalogs”. In this sense, the ACCESS-NRI catalog can be though of as providing a catalog of catalogs. To try and avoid confusion in these docs, we will use the term “datastore” instead of “catalog” when referring to Intake-ESM.

Note

The CMIP5 and CMIP6 Intake-ESM datastores generated by NCI are available as entries in the ACCESS-NRI catalog.

Aliasing and CMIP compatibility

The ACCESS-NRI catalog includes built-in aliasing support, which lets you search using alternative, user-friendly names that are automatically mapped to the underlying canonical values stored in the catalog. This is particularly useful for researchers familiar with CMIP vocabularies who want to discover raw ACCESS model outputs without needing to learn ACCESS-specific variable codes or column names.

For example, you can search for variable="tas" and the catalog will find files stored under the raw ACCESS code fld_s03i236 as well as any files already labelled tas. Similarly, frequency="daily" will match entries stored as 1day, and source_id="ACCESS-ESM1-5" is accepted in place of the native model column name.

See also

Aliasing — full reference documentation for all supported aliases, field mappings, and CMIP-to-ACCESS variable tables.

Aliasing demo — interactive Jupyter notebook demonstrating aliasing on Gadi.

• pr (precipitation) → precip

When a mapping occurs, you’ll see a helpful message showing what translation was applied, so you can learn the ACCESS variable names as you work.

Benefits:

• Use familiar CMIP terminology to find ACCESS data

• No need to learn ACCESS-specific variable names upfront

• Transparent - you can see what mappings are applied

• Backwards compatible - existing workflows continue to work

A set of core metadata attributes are associated with each entry in the ACCESS-NRI catalog. At the moment these include:

• The name of the data product

• A short description of the data product

• The model(s) used

• The realm(s) available

• The frequency(/ies) available

• The variable(s) available

A simple search API allows users to filter the entries in the catalog based on these metadata attributes. The idea is that users will:

1. search the ACCESS-NRI catalog for data products, e.g. products containing the models, variables etc that are of interest to them.

2. open the Intake-ESM datastore(s) for the product(s) of interest.

3. search the Intake-ESM datastore(s) for the datasets within each product that are of interest to them. A “dataset” here is a set of files that can be readily opened and combined for analysis.

4. open the datasets of interest as xarray Dataset(s).

5. perform some analysis on the xarray Dataset(s).