.. _what: What is it? =========== .. note:: These docs assume readers have a basic knowledge of Python and are familiar with the Python package, xarray. If you're not familiar with xarray, check out the excellent `xarray tutorial webpage `_. The short, high-level description ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The ACCESS-NRI catalog is essentially a table of climate data products that exist on Gadi. Each entry in the table corresponds to a different product, and the columns contain attributes associated with each product--things like the models, frequencies and variables available. Users can search on the attributes to find the products that might be useful to them. For example, a user might want to know which data products contain variables X, Y and Z at monthly frequency. The ACCESS-NRI catalog enables users to find products that satisfy their query and to subsequently load their data without having to know the location and structure of the underlying files. .. topic:: Hang on, I don't have access to Gadi! Never fear, you may still have a use for :code:`access-nri-intake-catalog`! You can use the :ref:`builders` in the package to create Intake-ESM datastores for your own experiments. See the `Quickstart tutorial`_ on building your own Intake-ESM datastores for details. .. _Quickstart tutorial: ../datastores/quickstart.html .. _what_detailed: The more detailed description ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The ACCESS-NRI catalog provides a catalog of *Intake sources* and associated metadata. .. topic:: Wait, what are "Intake sources"? "Intake" is a Python package that provides a general interface for loading data in Python. Intake provides a set of data loaders (called "drivers") that allow users to load a wide range of data formats into familiar Python data structures (e.g. a pandas dataframe or xarray Dataset) using the exact same code. Some drivers are built into Intake, and some are provided by other "plugin" packages. Documentation of the Intake package can be found `here `_ and a list of available Intake drivers can be found `here `_. "Intake sources" describe data that can be loaded using an Intake driver. For example, an Intake source might describe a simple csv file to be opened using the inbuilt Intake "csv" driver, or it might describe a set of netcdf files to be opened using the plugin `Intake-ESM `_ "esm_datastore" driver. In fact, the ACCESS-NRI catalog itself is a type of Intake source that uses the plugin `Intake-dataframe-catalog `_ "df_catalog" driver. The entries in the ACCESS-NRI catalog are *Intake-ESM datastores* for climate data products that exist on Gadi. .. topic:: Wait, what are "Intake-ESM datastores"? Intake-ESM datastores are a type of Intake source that describes a climate data product comprising many files (e.g. netcdf files). Users can search across metadata associated with each file (e.g. a variable name) and open contiguous files into xarray Datasets for subsequent analysis. For example, NCI provides Intake-ESM datastores for the CMIP5 and CMIP6 data collections available on Gadi. In `their documentation `_ they demonstrate how to execute a search across these datastores to find and open a few datasets. Intake-ESM datastores are often also referred to as Intake-ESM "catalogs". In this sense, the ACCESS-NRI catalog can be though of as providing a catalog of catalogs. To try and avoid confusion in these docs, we will use the term "datastore" instead of "catalog" when referring to Intake-ESM. .. note:: The CMIP5 and CMIP6 Intake-ESM datastores generated by NCI are available as entries in the ACCESS-NRI catalog. .. _cmip_compatibility: Aliasing and CMIP compatibility ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The ACCESS-NRI catalog includes built-in **aliasing** support, which lets you search using alternative, user-friendly names that are automatically mapped to the underlying canonical values stored in the catalog. This is particularly useful for researchers familiar with CMIP vocabularies who want to discover raw ACCESS model outputs without needing to learn ACCESS-specific variable codes or column names. For example, you can search for ``variable="tas"`` and the catalog will find files stored under the raw ACCESS code ``fld_s03i236`` as well as any files already labelled ``tas``. Similarly, ``frequency="daily"`` will match entries stored as ``1day``, and ``source_id="ACCESS-ESM1-5"`` is accepted in place of the native ``model`` column name. .. seealso:: :ref:`aliasing` — full reference documentation for all supported aliases, field mappings, and CMIP-to-ACCESS variable tables. :doc:`aliases_demo` — interactive Jupyter notebook demonstrating aliasing on Gadi. * ``pr`` (precipitation) → ``precip`` When a mapping occurs, you'll see a helpful message showing what translation was applied, so you can learn the ACCESS variable names as you work. **Benefits:** * Use familiar CMIP terminology to find ACCESS data * No need to learn ACCESS-specific variable names upfront * Transparent - you can see what mappings are applied * Backwards compatible - existing workflows continue to work A set of core metadata attributes are associated with each entry in the ACCESS-NRI catalog. At the moment these include: * The name of the data product * A short description of the data product * The model(s) used * The realm(s) available * The frequency(/ies) available * The variable(s) available A simple search API allows users to filter the entries in the catalog based on these metadata attributes. The idea is that users will: #. search the ACCESS-NRI catalog for data products, e.g. products containing the models, variables etc that are of interest to them. #. open the Intake-ESM datastore(s) for the product(s) of interest. #. search the Intake-ESM datastore(s) for the datasets within each product that are of interest to them. A "dataset" here is a set of files that can be readily opened and combined for analysis. #. open the datasets of interest as xarray Dataset(s). #. perform some analysis on the xarray Dataset(s).