Add weekly build/test scripts

[AlexanderRichert-NOAA]

Summary

This PR will add scripting for weekly unit testing and build caching of the unified env through Jenkins, cron jobs, etc.

Also a few updates for Acorn, including a change to spack stack create env --compiler that allows compiler version to be specified (spack stack create env --compiler intel -> envs/myenv-intel; spack stack create env --compiler intel@2022 -> envs/myenv-intel-2022), which is needed to differentiate on platforms platform where more than one version of a compiler is installed.

Testing/implementation by platform

On each system:

• Configure ShellSetup.sh (including setting centralized build_cache dir under spack-stack/ dir)
• Set up centralized source cache
• Get it running (Jenkins, cron, GHA); test installing a few packages
• Enable notifications (email, Slack, whatever)

Platforms (check when all of the above items are complete):

• Acorn - @AlexanderRichert-NOAA - cron
• Orion - @rickgrubin-tomorrow test with Jenkins on Orion
• Nautilus - @climbfuji - either cron or Github actions

Applications affected

all

Systems affected

all (eventually)

Dependencies

none

Issue(s) addressed

#1184, #365

Checklist

• This PR addresses one issue/problem/enhancement, or has a very good reason for not doing so.
• These changes have been tested on the affected systems and applications.
• All dependency PRs/issues have been resolved and this PR can be merged.

[AlexanderRichert-NOAA]

closing temporarily so I don't spam the CI with jobs...

[AlexanderRichert-NOAA]

Cron entry for Acorn: 0 1 * * SUN /path/to/spack-stack-weekly-build/util/weekly_build/SpackStackBuildCache_AllSteps.sh $(date +\%y\%m\%d) /path/to/spack-stack/weekly_build acorn > /path/to/spack-stack/weekly_build/logs/cron.$(date +\%y\%m\%d).out 2>&1

[AlexanderRichert-NOAA]

I just noticed that on Acorn, it was creating a directory called "file:" in $SPACK_ENV because of the way it handles the mirror path. I'm tweaking it to drop the "file://" part when it looks up the directory; @rickgrubin-tomorrow @climbfuji please re-test with this change and make sure I didn't break anything...

[climbfuji]

I just noticed that on Acorn, it was creating a directory called "file:" in $SPACK_ENV because of the way it handles the mirror path. I'm tweaking it to drop the "file://" part when it looks up the directory; @rickgrubin-tomorrow @climbfuji please re-test with this change and make sure I didn't break anything...

Yes, that's fine. I never used file:// in the spack mirror create command.

[climbfuji]

I just pushed develop into this branch. Once we have approval from EPIC (Rick and or Bruce), we can merge this.

[AlexanderRichert-NOAA]

I need to make one or two last tweaks to the UWM test scripting then it'll be good to go on my end.

[AlexanderRichert-NOAA]

@rickgrubin-noaa I made a few small changes, mainly a couple fixes, plus I'm putting ufs-weather-model under the env directory. This works on Acorn, so if this looks good to you and I didn't break anything for Orion, I'd like to push it as is then later on refine things like having it modify ufs-weather-model modulefiles so we don't need a fork and that sort of thing. Thanks again for adding the UWM testing-- once I added in the shell setup call because of using cron, it ran smoothly.

[rickgrubin-noaa]

@rickgrubin-noaa I made a few small changes, mainly a couple fixes, plus I'm putting ufs-weather-model under the env directory. This works on Acorn, so if this looks good to you and I didn't break anything for Orion, I'd like to push it as is then later on refine things like having it modify ufs-weather-model modulefiles so we don't need a fork and that sort of thing. Thanks again for adding the UWM testing-- once I added in the shell setup call because of using cron, it ran smoothly.

All changes are fine with me; nothing breaks on orion. I wasn't testing from cron, thus didn't notice the shell setup call missing, sorry.

I had considered modifying ufs-weather-model files on the fly (as it's a couple of sed statements that could be defined within a function in sites/<site>.sh) but had already implemented the branch of UFS WM approach.

Please merge, thank you.