Re-work testing verification, fail file naming consistency, remove ls…

…f support, simplify RT logs. (#2061)

* UFSWM
  * Re-work test verification in rt.sh for developers to use in PR process. (Closes #2058)
  * fail_<test_number> file name upgraded to use compile/test name instead of number.
  * Regression test logs do not clearly indicate when a test fails to run. (Closes #1821)
  * New test_changes.list file:
    * Verification of what tests change for each PR.
    * Used with './rt.sh -c -b test_changes.list` to help create new baselines for each PR.

.github/pull_request_template.md

@@ -1,93 +1,144 @@
 <!-- INSTRUCTIONS: 
-- Please fill out all sections of this PR and complete the checklist below
-- Please be as descriptive as possible, this is really important.
-- Please "fill in" checkboxes. Use [X] for a filled in checkbox or leave it [ ] for an empty checkbox
-- Please use github markup as much as possible in linking
-i.e.:
-* Linking to UFSWM PR's and issues add - #<pr/issue number>
-* Linking to a subcomponent PR and issues add - <Group>/<Fork>/pull/<number> or - <Group>/<Fork>/issues/<number>
+- PLEASE READ/FOLLOW THE DIRECTIONS IN EACH SECTION
+- Complete the 'Commit Queue Requirements' below
+- Please use github markup as much as possible (https://docs.github.com/en/get-started/writing-on-github)
+- Please leave your PR in a draft state until all underlying work is completed.
 -->
 ## Commit Queue Requirements:
-<!-- Please note: PRs will be scheduled in the Commit Queue in the order received and only after all
-pre-requisite testing is complete and all PR requirements (e.g. Issues created and noted, Subcomponent
-PRs reviewed and accepted) are met. -->
+<!--
+- Please complete the items that follow this.
+- Please "check off" completed items. Use [X] for a filled in checkbox or leave it [ ] for an empty checkbox
+- Your PR will not be considered until all requirements are met.
+- THIS IS YOUR RESPONSIBILITY
+ -->
 - [ ] Fill out all sections of this template.
 - [ ] All sub component pull requests have been reviewed by their code managers.
-- [ ] Run the full RT suite (compared to current baselines) on either Hera/Derecho/Hercules AND have committed the log to my PR branch.
-- [ ] Add list of all failed regression tests in "Regression Tests" section.
-
-## PR Information
+- [ ] Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
+- [ ] Commit 'test_changes.list' from previous step
+---
+## Description:
+<!--
+Please provide a detailed verbose description of what this PR does
+-->
 
-### Description
-<!-- Provide a detailed description of what this PR does in the space provided below-->
 
-### Commit Message
+### Commit Message:
 <!--
-Please provide the following concise information:
-Description of all changes - 1 line
-Please list all individual issue titles addressed with github links at the end in parenthesis (using #<number> or <group>/<fork>/issues/<number>).
+Please provide concise information for The UFS-WM and/or each sub-component:
+Please delete what is not needed.
 -->
+```
+* UFSWM - 
+  * AQM - 
+  * CDEPS - 
+  * CICE - 
+  * CMEPS - 
+  * CMakeModules - 
+  * FV3 - 
+    * ccpp-physics - 
+    * atmos_cubed_sphere - 
+  * GOCART - 
+  * HYCOM - 
+  * MOM6 - 
+  * NOAHMP - 
+  * WW3 - 
+  * stochastic_physics - 
+```
 
+### Priority:
+<!--
+Please provide the priority you would prefer this pull request to have.
+* Critical Bugfix: Model is wrong.
+* High: Time-sensitive project.
+* Normal.
+Please delete the ones that are not applicable
+-->
+* Critical Bugfix: Reason
+* High: Reason
+* Normal
 
-### Priority
-- [ ] Critical Bugfix (This PR contains a critical bug fix and should be prioritized.)
-- [ ] High (This PR contains a feature or fix needed for a time-sensitive project (eg, retrospectives, implementations))
-- [ ] Normal
-
-### Blocking Dependencies
-<!-- If there are any PR's that are needed to be completed before this one, please add links
-     to them here -->
+## Git Tracking
+### UFSWM:
+<!--
+Please add the UFS-WM github issue here if there is one
+Please delete the one that is not applicable.
+-->
+* Closes #
+* None
 
-### Git Issues Fixed By This PR
-<!-- Example: - Closes #1698  or - Closes NOAA-EMC/fv3atm/issues/729 -->
+### Sub component Pull Requests:
+<!--
+Please provide a list of sub-components involved with this pull request.
+Please provide links to the sub-component pull requests as shown below.
+Please delete what is not needed.
+Example:
+* FV3: NOAA-EMC/fv3atm#734
+  * ccpp-physics: ufs-community/ccpp-physics#33
+* WW3: NOAA-EMC/WW3#321
+-->
+* AQM:
+* CDEPS:
+* CICE:
+* CMEPS:
+* CMakeModules:
+* FV3:
+  * ccpp-physics:
+  * atmos_cubed_sphere:
+* GOCART:
+* HYCOM:
+* MOM6:
+* NOAHMP:
+* WW3:
+* stochastic_physics:
+* None
 
+### UFSWM Blocking Dependencies:
+<!--
+If there are any UFSWM PR's that are needed to be completed before this one, please add links
+to them here
+Please delete what is not needed.
+-->
+* Blocked by #
+* None
 
+---
 ## Changes
-
-### Subcomponent (with links)
-<!-- (add links to subcomponent PR's here) -->
-<!-- Example:
-[X] FV3
-- NOAA-EMC/fv3atm/pull/734
-- NOAA-EMC/fv3atm/pull/735
+### Regression Test Changes (Please commit test_changes.list):
+<!--
+Please let us know if this PR creates new baselines, changes baselines or not.
+Please delete what is not needed.
+Please make sure you have properly submitted test_changes.list
 -->
-- [ ] AQM
-- [ ] CDEPS
-- [ ] CICE
-- [ ] CMEPS
-- [ ] CMakeModules
-- [ ] FV3
-- [ ] GOCART
-- [ ] HYCOM
-- [ ] MOM6
-- [ ] NOAHMP
-- [ ] WW3
-- [ ] stochastic_physics
-- [ ] none
+* PR Adds New Tests/Baselines.
+* PR Updates/Changes Baselines.
+* No Baseline Changes.
 
-### Input data
-- [ ] No changes are expected to input data.
-- [ ] Changes are expected to input data:
-  - [ ] New input data.
-  - [ ] Updated input data.
-
-### Regression Tests:
-- [ ] No changes are expected to any regression test.
-- [ ] Changes are expected to the following tests:
-<details><summary>FAILED REGRESSION TESTS</summary>
-<!-- List failed regression tests here or add "None" -->
-
-</details>
+### Input data Changes:
+<!--
+If there are any changes to input-data for a test, please provide information here.
+Please delete what is not needed.
+-->
+* None.
+* New input data.
+* Updated input data.
 
-### Libraries
-<!-- Library updates take time. If this PR needs updates to libraries, please make sure to accomplish the following tasks -->
-- [ ] Not Needed
-- [ ] Needed
-  - [ ] Create separate issue in [JCSDA/spack-stack](https://github.com/JCSDA/spack-stack) asking for update to library. Include library name, library version.
-  - [ ] Add issue link from JCSDA/spack-stack following this item <!-- for example: "- JCSDA/spack-stack/issue/1757" -->
+### Library Changes/Upgrades:
+<!-- Library updates take time. Please provide library and version information here.
+** SPECIAL INSTRUCTIONS **
+If this PR needs updates to libraries please make sure to accomplish the following tasks:
+- Create separate issue in (https://github.com/JCSDA/spack-stack) asking for update to library. Include library name, library version.
+- Add issue link from JCSDA/spack-stack following this item <!-- for example: "* JCSDA/spack-stack#1757"
 
+Please delete what is not needed.
+-->
+* Required
+  * Library names w/versions:
+  * Git Stack Issue (JCSDA/spack-stack#)
+* No Updates
+
+---
 <!-- STOP!!! THE FOLLOWING IS FOR CODE MANAGERS ONLY. PLEASE DO NOT FILL OUT -->
-### Testing Log:
+## Testing Log:
 - RDHPCS
   - [ ] Hera
   - [ ] Orion
@@ -98,8 +149,5 @@ Please list all individual issue titles addressed with github links at the end i
 - WCOSS2
   - [ ] Dogwood/Cactus
   - [ ] Acorn
-- CI
-  - [ ] Completed
-- opnReqTest
-  - [ ] N/A
-  - [ ] Log attached to comment
+- [ ] CI
+- [ ] opnReqTest (complete task if unnecessary)

modulefiles/ufs_derecho.intel.lua

@@ -3,7 +3,7 @@ loads UFS Model prerequisites for NOAA Parallelworks/Intel
 ]])
 
 setenv("LMOD_TMOD_FIND_FIRST","yes")
-prepend_path("MODULEPATH", "/lustre/desc1/scratch/epicufsrt/contrib/modulefiles")
+prepend_path("MODULEPATH", "/glade/work/epicufsrt/contrib/spack-stack/derecho/modulefiles")
 load("ecflow/5.8.4")
 load("mysql/8.0.33")
 

tests/compile.sh

@@ -24,20 +24,20 @@ fi
 readonly ARGC=$#
 
 if [[ $ARGC -lt 2 ]]; then
-  echo "Usage: $0 MACHINE_ID [ MAKE_OPT [ COMPILE_NR ] [ RT_COMPILER ] [ clean_before ] [ clean_after ] ]"
+  echo "Usage: $0 MACHINE_ID [ MAKE_OPT [ COMPILE_ID ] [ RT_COMPILER ] [ clean_before ] [ clean_after ] ]"
   echo Valid MACHINE_IDs:
   echo $( ls -1 ../cmake/configure_* | sed s:.*configure_::g | sed s:\.cmake:: ) | fold -sw72
   exit 1
 else
   MACHINE_ID=$1
   MAKE_OPT=${2:-}
-  COMPILE_NR=${3:+_$3}
+  COMPILE_ID=${3:+_$3}
   RT_COMPILER=${4:-intel}
   clean_before=${5:-YES}
   clean_after=${6:-YES}
 fi
 
-BUILD_NAME=fv3${COMPILE_NR}
+BUILD_NAME=fv3${COMPILE_ID}
 
 PATHTR=${PATHTR:-$( cd ${MYDIR}/.. && pwd )}
 BUILD_DIR=${BUILD_DIR:-$(pwd)/build_${BUILD_NAME}}
@@ -131,4 +131,4 @@ fi
 
 elapsed=$SECONDS
 echo "Elapsed time $elapsed seconds. Compiling ${CMAKE_FLAGS} finished"
-echo "Compile ${COMPILE_NR/#_} elapsed time $elapsed seconds. ${CMAKE_FLAGS}" > compile${COMPILE_NR}_time.log
+echo "Compile ${COMPILE_ID/#_} elapsed time $elapsed seconds. ${CMAKE_FLAGS}" > compile${COMPILE_ID}_time.log

tests/fv3_conf/compile_qsub.IN_acorn

@@ -15,7 +15,7 @@ cd $PBS_O_WORKDIR
 echo -n " $( date +%s )," >  job_timestamp.txt
 echo "Compile started:  " `date`
 
-@[PATHRT]/compile.sh @[MACHINE_ID] "@[MAKE_OPT]" @[COMPILE_NR] @[RT_COMPILER]
+@[PATHRT]/compile.sh @[MACHINE_ID] "@[MAKE_OPT]" @[COMPILE_ID] @[RT_COMPILER]
 
 echo "Compile ended:    " `date`
 echo -n " $( date +%s )," >> job_timestamp.txt

tests/fv3_conf/compile_qsub.IN_derecho

@@ -13,7 +13,7 @@ echo -n " $( date +%s )," >  job_timestamp.txt
 echo "Compile started:  " `date`
 module purge
 
-@[PATHRT]/compile.sh @[MACHINE_ID] "@[MAKE_OPT]" @[COMPILE_NR] @[RT_COMPILER]
+@[PATHRT]/compile.sh @[MACHINE_ID] "@[MAKE_OPT]" @[COMPILE_ID] @[RT_COMPILER]
 
 echo "Compile ended:    " `date`
 echo -n " $( date +%s )," >> job_timestamp.txt

tests/fv3_conf/compile_qsub.IN_wcoss2

@@ -15,7 +15,7 @@ cd $PBS_O_WORKDIR
 echo -n " $( date +%s )," >  job_timestamp.txt
 echo "Compile started:  " `date`
 
-@[PATHRT]/compile.sh @[MACHINE_ID] "@[MAKE_OPT]" @[COMPILE_NR] @[RT_COMPILER]
+@[PATHRT]/compile.sh @[MACHINE_ID] "@[MAKE_OPT]" @[COMPILE_ID] @[RT_COMPILER]
 
 echo "Compile ended:    " `date`
 echo -n " $( date +%s )," >> job_timestamp.txt

tests/fv3_conf/compile_slurm.IN_gaea

@@ -15,7 +15,7 @@ set -eux
 echo -n " $( date +%s )," >  job_timestamp.txt
 echo "Compile started:  " `date`
 
-@[PATHRT]/compile.sh @[MACHINE_ID] "@[MAKE_OPT]" @[COMPILE_NR] @[RT_COMPILER]
+@[PATHRT]/compile.sh @[MACHINE_ID] "@[MAKE_OPT]" @[COMPILE_ID] @[RT_COMPILER]
 
 echo "Compile ended:    " `date`
 echo -n " $( date +%s )," >> job_timestamp.txt

tests/fv3_conf/compile_slurm.IN_hera

@@ -13,7 +13,7 @@ set -eux
 echo -n " $( date +%s )," >  job_timestamp.txt
 echo "Compile started:  " `date`
 
-@[PATHRT]/compile.sh @[MACHINE_ID] "@[MAKE_OPT]" @[COMPILE_NR] @[RT_COMPILER]
+@[PATHRT]/compile.sh @[MACHINE_ID] "@[MAKE_OPT]" @[COMPILE_ID] @[RT_COMPILER]
 
 echo "Compile ended:    " `date`
 echo -n " $( date +%s )," >> job_timestamp.txt

tests/fv3_conf/compile_slurm.IN_hercules

@@ -14,7 +14,7 @@ set -eux
 echo -n " $( date +%s )," >  job_timestamp.txt
 echo "Compile started:  " `date`
 
-@[PATHRT]/compile.sh @[MACHINE_ID] "@[MAKE_OPT]" @[COMPILE_NR] @[RT_COMPILER]
+@[PATHRT]/compile.sh @[MACHINE_ID] "@[MAKE_OPT]" @[COMPILE_ID] @[RT_COMPILER]
 
 echo "Compile ended:    " `date`
 echo -n " $( date +%s )," >> job_timestamp.txt

tests/fv3_conf/compile_slurm.IN_jet

@@ -14,7 +14,7 @@ set -eux
 echo -n " $( date +%s )," >  job_timestamp.txt
 echo "Compile started:  " `date`
 
-@[PATHRT]/compile.sh @[MACHINE_ID] "@[MAKE_OPT]" @[COMPILE_NR] @[RT_COMPILER]
+@[PATHRT]/compile.sh @[MACHINE_ID] "@[MAKE_OPT]" @[COMPILE_ID] @[RT_COMPILER]
 
 echo "Compile ended:    " `date`
 echo -n " $( date +%s )," >> job_timestamp.txt

tests/fv3_conf/compile_slurm.IN_noaacloud

@@ -11,7 +11,7 @@ set -eux
 echo -n " $( date +%s )," >  job_timestamp.txt
 echo "Compile started:  " `date`
 
-@[PATHRT]/compile.sh @[MACHINE_ID] "@[MAKE_OPT]" @[COMPILE_NR]
+@[PATHRT]/compile.sh @[MACHINE_ID] "@[MAKE_OPT]" @[COMPILE_ID]
 
 echo "Compile ended:    " `date`
 echo -n " $( date +%s )," >> job_timestamp.txt

tests/fv3_conf/compile_slurm.IN_orion

@@ -14,7 +14,7 @@ set -eux
 echo -n " $( date +%s )," >  job_timestamp.txt
 echo "Compile started:  " `date`
 
-@[PATHRT]/compile.sh @[MACHINE_ID] "@[MAKE_OPT]" @[COMPILE_NR] @[RT_COMPILER]
+@[PATHRT]/compile.sh @[MACHINE_ID] "@[MAKE_OPT]" @[COMPILE_ID] @[RT_COMPILER]
 
 echo "Compile ended:    " `date`
 echo -n " $( date +%s )," >> job_timestamp.txt

tests/fv3_conf/compile_slurm.IN_s4

@@ -13,7 +13,7 @@ set -eux
 echo -n " $( date +%s )," >  job_timestamp.txt
 echo "Compile started:  " `date`
 
-@[PATHRT]/compile.sh @[MACHINE_ID] "@[MAKE_OPT]" @[COMPILE_NR] @[RT_COMPILER]
+@[PATHRT]/compile.sh @[MACHINE_ID] "@[MAKE_OPT]" @[COMPILE_ID] @[RT_COMPILER]
 
 echo "Compile ended:    " `date`
 echo -n " $( date +%s )," >> job_timestamp.txt