Building sources

Each entry in an intake-dataframe-catalog refers to an Intake source. Sources may be pre-generated (e.g. the CMIP5 and CMIP6 Intake-ESM datastore sources managed by NCI) or they can be built using the access_nri_intake.source sub-package. Currently, access_nri_intake.source only supports building Intake-ESM datastore sources.

When access_nri_intake.source is first imported, it downloads and parses a specific commit of the schema at ACCESS-NRI/schema and stores it in the variable access_nri_intake.source.ESM_JSONSCHEMA. This schema defines what metadata must be included in an Intake-ESM catalog, and what types and fields are allowed. Subsequent imports read the downloaded schema, unless the schema is changed (see Catalog schema), in which case the new schema is downloaded.

Note

The “required” field in the downloaded schema is replaced with access_nri_intake.source.CORE_COLUMNS prior to saving to access_nri_intake.source.ESM_JSONSCHEMA to allow this field to be customized within access-nri-intake

Builders

New Intake-ESM datastores can be created for ACCESS model output using the Builder classes in the sub-module access_nri_intake.source.builders. All public Builders inherit from access_nri_intake.source.builders.BaseBuilder, which inherits from ecgtools.builder.Builder. access_nri_intake.source.builders.BaseBuilder simply reorganises the functionality in ecgtools.builder.Builder so that self-contained Builder classes can be easily written for different types of model output. All Builders validate against the schema in access_nri_intake.source.ESM_JSONSCHEMA.

Creating a new Builder

New Builders should inherit from access_nri_intake.source.builders.BaseBuilder. In general, all that should be required is to:

• Hardcode some of the optional kwargs on BaseBuilder.__init__ in the __init__ method of the new Builder. The exact choice of which kwargs are hardcoded will depend on the specifics of the target model output. It might be useful to look at existing Builder class implementations and at the ecgtools.builder.Builder class, which is inhereted by access_nri_intake.source.builders.BaseBuilder - see the ecgtools documentation here. The Intake-ESM specifications will also be helpful for interpreting many of the kwargs

• Overwrite the static parser method. This method should receive a single file (usually netcdf) and return a dictionary containing all of the metadata to include in the Intake-ESM datastore. As described above the dictionary is validated against access_nri_intake.source.ESM_JSONSCHEMA. The parser method serves the same function as custom parsers in ecgtools, so the documentation on these may be useful. Again, it will also be helpful to look at existing Builder class implementations.

API for access_nri_intake.source

This documentation has been auto-generated using sphinx-autoapi

• access_nri_intake.source
  • Submodules
    • access_nri_intake.source.builders
      • Classes
      • Module Contents
        • AccessOm2Builder
          • AccessOm2Builder.PATTERNS
          • AccessOm2Builder.parser()
          • AccessOm2Builder.TIME_PARSER
          • AccessOm2Builder.paths
          • AccessOm2Builder.depth
          • AccessOm2Builder.exclude_patterns
          • AccessOm2Builder.include_patterns
          • AccessOm2Builder.data_format
          • AccessOm2Builder.groupby_attrs
          • AccessOm2Builder.aggregations
          • AccessOm2Builder.storage_options
          • AccessOm2Builder.joblib_parallel_kwargs
          • AccessOm2Builder.parse()
          • AccessOm2Builder.save()
          • AccessOm2Builder.validate_parser()
          • AccessOm2Builder.build()
          • AccessOm2Builder.columns_with_iterables
          • AccessOm2Builder.parse_filename()
          • AccessOm2Builder.parse_ncfile()
          • AccessOm2Builder.get_assets()
          • AccessOm2Builder.clean_dataframe()
        • AccessOm3Builder
          • AccessOm3Builder.PATTERNS
          • AccessOm3Builder.parser()
          • AccessOm3Builder.TIME_PARSER
          • AccessOm3Builder.paths
          • AccessOm3Builder.depth
          • AccessOm3Builder.exclude_patterns
          • AccessOm3Builder.include_patterns
          • AccessOm3Builder.data_format
          • AccessOm3Builder.groupby_attrs
          • AccessOm3Builder.aggregations
          • AccessOm3Builder.storage_options
          • AccessOm3Builder.joblib_parallel_kwargs
          • AccessOm3Builder.parse()
          • AccessOm3Builder.save()
          • AccessOm3Builder.validate_parser()
          • AccessOm3Builder.build()
          • AccessOm3Builder.columns_with_iterables
          • AccessOm3Builder.parse_filename()
          • AccessOm3Builder.parse_ncfile()
          • AccessOm3Builder.get_assets()
          • AccessOm3Builder.clean_dataframe()
        • Mom6Builder
          • Mom6Builder.PATTERNS
          • Mom6Builder.TIME_PARSER
          • Mom6Builder.parser()
          • Mom6Builder.paths
          • Mom6Builder.depth
          • Mom6Builder.exclude_patterns
          • Mom6Builder.include_patterns
          • Mom6Builder.data_format
          • Mom6Builder.groupby_attrs
          • Mom6Builder.aggregations
          • Mom6Builder.storage_options
          • Mom6Builder.joblib_parallel_kwargs
          • Mom6Builder.parse()
          • Mom6Builder.save()
          • Mom6Builder.validate_parser()
          • Mom6Builder.build()
          • Mom6Builder.columns_with_iterables
          • Mom6Builder.parse_filename()
          • Mom6Builder.parse_ncfile()
          • Mom6Builder.get_assets()
          • Mom6Builder.clean_dataframe()
        • AccessEsm15Builder
          • AccessEsm15Builder.PATTERNS
          • AccessEsm15Builder.parser()
          • AccessEsm15Builder.TIME_PARSER
          • AccessEsm15Builder.paths
          • AccessEsm15Builder.depth
          • AccessEsm15Builder.exclude_patterns
          • AccessEsm15Builder.include_patterns
          • AccessEsm15Builder.data_format
          • AccessEsm15Builder.groupby_attrs
          • AccessEsm15Builder.aggregations
          • AccessEsm15Builder.storage_options
          • AccessEsm15Builder.joblib_parallel_kwargs
          • AccessEsm15Builder.parse()
          • AccessEsm15Builder.save()
          • AccessEsm15Builder.validate_parser()
          • AccessEsm15Builder.build()
          • AccessEsm15Builder.columns_with_iterables
          • AccessEsm15Builder.parse_filename()
          • AccessEsm15Builder.parse_ncfile()
          • AccessEsm15Builder.get_assets()
          • AccessEsm15Builder.clean_dataframe()
        • AccessCm2Builder
          • AccessCm2Builder.PATTERNS
          • AccessCm2Builder.parser()
          • AccessCm2Builder.TIME_PARSER
          • AccessCm2Builder.paths
          • AccessCm2Builder.depth
          • AccessCm2Builder.exclude_patterns
          • AccessCm2Builder.include_patterns
          • AccessCm2Builder.data_format
          • AccessCm2Builder.groupby_attrs
          • AccessCm2Builder.aggregations
          • AccessCm2Builder.storage_options
          • AccessCm2Builder.joblib_parallel_kwargs
          • AccessCm2Builder.parse()
          • AccessCm2Builder.save()
          • AccessCm2Builder.validate_parser()
          • AccessCm2Builder.build()
          • AccessCm2Builder.columns_with_iterables
          • AccessCm2Builder.parse_filename()
          • AccessCm2Builder.parse_ncfile()
          • AccessCm2Builder.get_assets()
          • AccessCm2Builder.clean_dataframe()
    • access_nri_intake.source.utils
      • Attributes
      • Exceptions
      • Classes
      • Module Contents
        • FREQUENCY_STATIC
        • EmptyFileError
        • GenericTimeParser
          • GenericTimeParser.TIMEINFO_TIME_FORMAT
          • GenericTimeParser.ds
          • GenericTimeParser.filename_frequency
          • GenericTimeParser.time_dim
        • AccessTimeParser
          • AccessTimeParser.TIMEINFO_TIME_FORMAT
          • AccessTimeParser.ds
          • AccessTimeParser.filename_frequency
          • AccessTimeParser.time_dim
        • GfdlTimeParser
          • GfdlTimeParser.ds
          • GfdlTimeParser.filename_frequency
          • GfdlTimeParser.time_dim
          • GfdlTimeParser.TIMEINFO_TIME_FORMAT
  • Attributes
  • Package Contents
    • CORE_COLUMNS
    • PATH_COLUMN
    • VARIABLE_COLUMN