Releases

Releases#

Preparing a new code release#

New releases of access-nri-intake to PyPI and conda are published automatically when a tag is pushed to Github. A new release may or may not include an update to the ACCESS-NRI catalog files on Gadi.

  1. Go to ACCESS-NRI/access-nri-intake-catalog

  2. Click on “Releases”->”Draft new release” on the right-hand side of the screen

  3. Enter the new version (vX.X.X) as the tag and release title. Add a brief description of the release.

    Note

    It is recommended to attempt a beta release before committing to a major code update. In this case, the version number requires an ordinal after the b, e.g., v1.2.3b0. If the ordinal isn’t provided, the GitHub PyPI build action will append one, which breaks the linkage between the PyPI and Conda build actions.

  4. Click on “Publish release”. This should create the release on GitHub and trigger the workflow that builds and uploads the new version to PyPI and conda

Alternatively (though discouraged), one can trigger the new release from the command line:

$ export RELEASE=vX.X.X
$ git fetch --all --tags
$ git commit --allow-empty -m "Release $RELEASE"
$ git tag -a $RELEASE -m "Version $RELEASE"
$ git push --tags

Generating a new catalog version#

  1. Create a new version of the catalog on Gadi (this will take about 2 hours):

    $ export RELEASE=vYYYY-MM-DD
$ cd bin
$ qsub -v version=${RELEASE} build_all.sh

    Note

    Running the build script requires access to an up-to-date checkout of the access-nri-intake-catalog repository. The default location for this is /g/data/xp65/admin/access-nri-intake-catalog. If you do not have the ability to update this checkout, you may use a local one; however, you will need to update the CONFIG_DIR variable in bin/build_all.sh to point at your checkout location.

    Note

    If version is not provided, the default used is the current date, in the format vYYYY-MM-DD. This should be acceptable in most cases.

    Note

    If you wish to perform a new catalog build without updating the default catalog version, you can use the --no-concrete flag. This will create and save a new catalog version, but leave it in a folder named .$VERSION in the specified catalog build location. To subsequently concretize this build, you can use the catalog-concretize command. Instructions for how to concretize the build will be available in the output of the build script.

  2. Updating access_nri_intake_catalog is no longer necessary - the new catalog will be available immediately as intake.cat.access_nri.

  3. Run the Jupyter notebook bin/new-build-checks.ipynb. This confirms the catalog versions that are available, and runs a comparison between the new catalog and a selected previous catalog for additions, deletions, etc. Verify that there are no unexpected changes in the catalog composition.

Concretizing new catalog builds#

If you have built a new catalog without concretizing it (either by using the --no-concrete flag or due to releasing two new catalog builds with the same version number), you can concretize the new catalog build by running the catalog-concretize command. This will concretize the specified catalog verison:

$ catalog-concretize –help usage: catalog-concretize [-h] [–build_base_path BUILD_BASE_PATH] [–version VERSION] [–catalog_file CATALOG_FILE]

[–catalog_base_path CATALOG_BASE_PATH] [–no_update] [–force]

Concretize a build by moving it to the final location and updating the paths in the catalog.json files.

options:
-h, --help

show this help message and exit

--build_base_path BUILD_BASE_PATH

The base path for the build.

--version VERSION

The version of the build.

--catalog_file CATALOG_FILE

The name of the catalog file.

--catalog_base_path CATALOG_BASE_PATH

The base path for the catalog. If None, the catalog_base_path will be set to the build_base_path. Defaults to None.

--no_update

Set this if you don’t want to update the catalog.yaml file. Defaults to False. If False, the catalog.yaml file will be updated.

--force

Force the concretization of the build, even if a version of the catalog with the specified version number already exists in the catalog_base_path. Defaults to False.

Running catalog-build with the --no_concretize flag will return a specification of how to concretize the build in its output. Similarly, whilst attempting to concretize a build with a previously existing version number will fail (unless --force is set), the error message will contain the correct command to concretize the build.

New release with new catalog#

In the case of a linked release of a new major access-nri-intake-catalog and a new catalog build, the recommened process is:

  1. Create a beta release of access-nri-intake-catalog;

  2. Use the beta release to build a new catalog;

  3. Iterate over the above steps until the desired result is achieved;

  4. Make a definitive code release.

Mirroring the catalog to Nectar object storage#

After a new catalog version has been built and concretized on Gadi, it can be mirrored to Nectar object storage using the mirror-to-cloud command:

$ mirror-to-cloud --catalog-version vYYYY-MM-DD

This copies the catalog parquet files from Gadi, restructures them for efficient web access, and uploads them to the access-nri-intake-catalog container in Nectar object storage.

Getting access to Nectar#

To run mirror-to-cloud you need credentials for the Nectar Cloud project that hosts the access-nri-intake-catalog object storage container.

  1. Log in to the Nectar Dashboard.

  2. Agree to the Nectar Terms and Conditions if prompted.

  3. Note your username — it is the email address shown in the top-right corner of the dashboard after login.

  4. Provide that email address to one of the tenant managers below so they can add you to the project:

    • Jo Basevi

    • Aidan Heerdegen

    • Romain Beucher

Once you have been added, generate application credentials for the project and place them in ~/.config/openstack/clouds.yaml. The default template names the cloud openstack — rename it to nectar to match the configuration expected by mirror-to-cloud.