Merge branch 'testing'

README.new

@@ -144,7 +144,7 @@ Lmod 8.7+
               Namely print the location of the module.
             * Support for "module --terse spider phdf5/1.12.2" that just prints the module in the hierarchy.
             * Move almost almost all cosmic:value() calls to inside function not at routine top-level.
-            * PR: #702: Report build time in UTC when $SOURCE_DATE_EPOCH is set
+            * PR #702: Report build time in UTC when $SOURCE_DATE_EPOCH is set
             * PR #679: support for downstream conflicts added
             * PR #679: support for dynamic LMOD_MODULERC
    (8.7.39) * Fix busted test for ModuleA.
@@ -156,6 +156,5 @@ Lmod 8.7+
             * ModuleTable modification: Changes to $MODULEPATH are stored in an entry.
    (8.7.42) * Fix setting of $FPATH when running bash or ksh shell and executing zsh -l.
    (8.7.43) * Support for ksh flag not needed anymore
-
-
-
+W.I.P:
+   (8.7.44) * Issue #710: path2pathA() keep double delims, Reduce triple delims to double

docs/source/025_new.rst

@@ -1,8 +1,17 @@
 New Features in Lmod
 ====================
 
+**MODULES_AUTO_DEFAULT**
+   (Lmod 8.7.41+) When this variable is set then Lmod treats *prereq*
+   as *depends_on* and *prereq_any* as *depends_on_any*.
+
+**depends_on_any("pkgA*,"pkgB",...)**
+   (Lmod 8.7.41+) A new function *depends_on_any()* which works like
+   *depends_on()* except that Lmod picks the first available in the
+   list. 
+
 **hook.register(<hook_name>, func, <action>)**
-   (Lmod 8.7.25) The hook.register function now takes a optional third
+   (Lmod 8.7.25+) The hook.register function now takes a optional third
    argument: action.  The legal actions are the followning strings:
    "replace", "append", "prepend".  See
    :ref:`registering_multiple_hook_functions-label` for more details.

docs/source/090_configuring_lmod.rst

@@ -244,6 +244,11 @@ the configuration option which will set the action.
   ~/.config/lmod but will check both directories when reading picking
   the most recent one.
 
+**MODULES_AUTO_HANDLING**:
+  [yes/no, default: no, --with-modulesAutoHandling].  
+  When this is set, Lmod will treat *prereq()* like *depends_on()* and
+  *prereq_any()* like *depends_on_any()*.
+
 Configuration or cosmic:assign() at startup
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

docs/source/098_dependent_modules.rst

@@ -7,7 +7,7 @@ Let's assume that module "X" depends on module "A". There are several
 ways to handle module dependency.  Inside the "X" modulefile you could
 have one of the following choices:
 
-#. Use ``depends_on("A","B")``
+#. Use ``depends_on("A","B")`` or ``depends_on_any("C","D", ...)``
 #. Use ``prereq("A")``
 #. Use ``load("A")``
 #. Use ``always_load("A")``
@@ -34,6 +34,17 @@ on "A" then::
    $ module purge; module load X Y; module unload X   => keep A   
    $ module purge; module load X Y; module unload X Y => unload A
 
+
+``depends_on_any("C","D")``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+The function ``depends_on_any("C","D")`` works similarly to
+``depends_on()`` except that Lmod picks the first available module
+listed.  It first checks to see if any of the modules are already
+loaded. If none are already loaded then picks the first one that can
+be loaded.  On unload, Lmod remembers which of the choices it loaded
+to unload.  It is not an error if that module has already been
+unloaded.
+
 Complex uses of ``depends_on()``
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

docs/source/250_site_package.rst

@@ -37,7 +37,7 @@ Then inside your /home/user/Lmod/SitePackage.lua do::
 
     require("Site")
 
-Each require statement can only ``require'' one name.  So make sure
+Each require statement can only ``require`` one name.  So make sure
 that you symlink to a new name in your personal SitePackage.lua directory.
 
 Checking if you have setup SitePackage.lua correctly

docs/source/410_Lmod_principals.rst

@@ -6,6 +6,34 @@ Module Naming
 FullName -> shortname/version
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+N/V vs. C/N/V
+~~~~~~~~~~~~~
+Lmod supports names like gcc/10.1 where gcc is the shortname and 10.1
+is the version.  It also supports C/N/V which is
+category/name/version.  So a module can be named compiler/gcc/10.1
+where the shortname is compiler/gcc and the version is 10.1.  The
+number of categories can be as many as a site wants to use.  It is
+just more typing.  Internally, Lmod only separates the fullName into a
+shortname (sn) and a version.
+
+N/V vs. N/V/V
+~~~~~~~~~~~~~
+
+Starting with version 7+, Lmod support two or more levels of versions
+(namely N/V/V). So sites might name a module gcc/x86_64/10.1, where the
+shortname is gcc and the version is x86_64/10.1. The depth of version
+is unlimited as is the number of names. So a site might name a module:
+compiler/gcc/x86_64/10.1 where the shortname is compiler/gcc and the
+version is x86_64/10.1
+
+See the discussion about consequences of using N/V/V vs. N/V in ...
+
+
+
+
+
+
+
 One Name Rule
 ~~~~~~~~~~~~~
 It is really the one shortname (or sn) rule.  A user can only load one
@@ -45,7 +73,7 @@ Encoding text strings can be converted to base64 encoding.  This
 means that all spaces and quotes are hidden from shell interpretation.
 
 The ModuleTable is encoded as:
-_ModuleTable001_ and _ModuleTable_Sz_
+_ModuleTable_Sz_ and _ModuleTable001_ _ModuleTable002_ ...
 
 Other environment variables used by Lmod internally all start with
 __LMOD_
@@ -56,6 +84,130 @@ Lmod communicates with users via a group of env. vars
 Thing like LMOD_CMD and LMOD_DIR etc.
 
 
+Lmod coding conventions
+~~~~~~~~~~~~~~~~~~~~~~~
+
+Notes
+~~~~~
+
+A discussion on the design of Lmod:
+
+* At the outermost level.  The lmod program produces text. This text
+  is evaluated by the appropriate tool. In other words, Lmod can
+  produce shell commands to be evaluated by a shell. It can also
+  produce python, perl or cmake commands to be evaluate by the
+  matching tool.
+
+* Internally, Lmod has a table of key-value pairs that contain
+  typically environment variables and their values.  If no errors are
+  encountered, then Lmod loops over the keys in alphabetical order and
+  generates the requested style for output (shell, python, ...).
+
+* Give an example modulefile:
+
+      $ cat acme/1.1.lua
+
+      setenv(       "SITE_ACME_DIR",   "/opt/apps/acme/1.1")
+      prepend_path( "PATH",            "/opt/apps/acme/1.1/bin")
+      prepend_path( "LD_LIBRARY_PATH", "/opt/apps/acme/1.1/bin")
+
+* $ module load acme
+
+* Lmod gets the subcommand "load" and converts it to the "Load" (via src/lmod.in.lua)
+  function in cmdfuncs.lua 
+
+  Note all subcommands from the command line map to functions in src/cmdfuncs.lua
+
+* Steps to load the module:
+** Find the modulefile: acme/1.1.lua
+** 1st Lmod walks the directories with the src/DirTree.lua to build
+   DirT table.  This table contains the directory tree.
+** 2nd: Lmod converts dirT into moduleA
+** The conversion into moduleA applies all the rules that Lmod
+   requires to know the shortname and version.
+*** Whether a module is a meta module (i.e. no version)
+*** Whether it is a N/V or N/V/V
+*** Any marked defaults in the directory tree (as oppose to
+    LMOD_MODULERC)
+* if LMOD_CACHED_LOAD is set then Lmod skips all those step because it
+  has already found moduleT ( the file is called spiderT but it is
+  really moduleA)
+* If all modules are in the form of N/V (instead of N/V/V) then
+  moduleT with multiple directories in the module path are joined into
+  one structure called LocationT
+
+* Go to spec/*/*_spec.lua to see an example of what the structure looks
+  like.  For exmaple spec/DirTree/DirTree_spec.lua for what dirT looks
+  like and spec/ModuleA/ModuleA_spec.lua
+
+* When a module is loaded.  All is known is the userName. That is the
+  name that the module name on the command line. It could be the
+  fullName (i.e. shortName/version) or just the shortName.
+
+* That userName is used to initialize on MName object.  This object is
+  used to convert the userName into the fileName.  This is a lazy
+  evaluation. The conversion from a userName into a fullName and
+  fileName is only done when it is needed.
+
+* This is because **module load compiler mpi* the mpi modulefiles
+  location might not be known until the compiler module has been
+  loaded.
+
+
+
+------------------------------------------------------------------------
+Outline of steps to load a modulefile
+* command line is parsed into a sub-command such as load
+* the load string is converted to an action -> Load_Usr
+** Note that all commands are found in src/cmdfuncs.lua
+* The operation of Load_Usr and l_usrLoad() are discussed here
+* Each positional argument is now a module to load (or unload)
+* Each userName of a module is converted to an MName object (and is
+  discussed here)
+* Each Lmod function inside a modulefile gets evaluated differently
+  depending on the mode.  When loading a module, the setenv() function
+  sets an environment variable.  When unloading a module, the setenv()
+  function unsets or clears the environment variable.
+
+  The various ways that Lmod evaluates its functions is controlled by
+  the MainControl base class and the derived (or inherited) classes
+  such as src/MC_Load.lua for module loads and src/MC_Unload.lua
+
+* So the command load calls Load_Usr() in src/cmdfuncs.lua.  That
+  calls l_usrLoad().  This then calls MainControl:load_usr() which
+  calls MainControl:load().  This then calls Hub:load() which is doing
+  the real work of loading a module
+
+* Hub:load() enforces the rules of Lmod
+
+* Lmod uses the MName object to convert the userName into a filename.
+
+* Then the function loadModuleFile() tells Lua to evaluate the
+  modulefile.  It does so by reading the entire contents of the file
+  into a string.  This string is given to Lua to evaluate inside a
+  sandbox (see details here)
+* A tcl modulefile is converted to a lua via tcl2lua.tcl
+
+* When lua encounters a function like setenv() or prepend_path(),
+  these functions are Lmod functions.
+
+* A function like setenv() and prepend_path() are found in src/modfuncs.lua.  
+  These functions check that the arguments are valid.  Then
+  mcp:setenv() is called.
+
+* Review what mcp:setenv() etc means
+
+* functions like MainControl:setenv() and MainControl:prepend_path()
+  sets a key-value pair in the varT table.
+
+* Explain what frameStk does
+
+* Once all modulefile(s) are evaluated, control returns to lmod.
+  Assuming no errors were encountered, then lmod generates text from
+  the key-value pairs stored in the varT table for the appropriate
+  "shell".  This includes the Moduletable, LOADEDMODULES and _LMFILES_
+
+* Hooks?
 
 
 

init/lmod_bash_completions

@@ -127,7 +127,7 @@ _module() {
         --quiet --terse --version --default --width -r --regexp --mt --latest \
         -I --ignore_cache --gitversion --dumpversion --config --miniConfig    \
         --timer -f --force --redirect --show_hidden -T --trace --nx           \
-        --no_extensions"
+        --no_extensions --loc --location --terse_show_extensions --pod "
 
   case "${prev}" in
     add|load|try-load)
@@ -193,7 +193,8 @@ _ml() {
   opts="-d -D -h -q -t -v -w -s --style --expert --quiet --help -H -f --force     \
         --quiet --terse --version --default --Verbose --width -r --regexp --mt    \
         --gitversion --ignore_cache -I --latest --pin_version --trace -T --config \
-        --miniConfig --config_json --raw --show_hidden"
+        --miniConfig --config_json --raw --show_hidden --loc --location           \
+        --terse_show_extensions --pod"
 
   case "${prev}" in
     rm|remove|unload|switch|swap)

init/zsh/_ml

@@ -12,9 +12,9 @@ _ml()
     '(- *)'{--version,-v}'[module command version]'
     '(-D --debug)'{--debug,-D}'[Send program tracing information to stderr]'
     '(-I --ignore_cache)'{--ignore_cache,-I}'[Treat the cache file(s) as out-of-date]'
-    '(-T --trace)'{--style,-s}'[Report major internal actions by Lmod]'
+    '(-T --trace)'{--trace,-T}'[Report major internal actions by Lmod]'
     '(-d --default)'{--default,-d}'[List default modules only when used with avail]'
-    '(-f --force)'{--force,-f}'Force removal of a sticky module or save an empty collection]'
+    '(-f --force)'{--force,-f}'[Force removal of a sticky module or save an empty collection]'
     '(-q --quiet)'{--quiet,-q}'[disable verbose messages]'
     '(-r --regexp)'{--regexp,-r}'[lua regular expression matching]'
     '(-s --style)'{--style,-s}'[avail output style]'
@@ -23,18 +23,21 @@ _ml()
     '--Verbose[show generated module command]'
     '--config[display full Lmod configuration]'
     '--config_json[display full Lmod configuration in json format]'
-    '--force[force the removal of a sticky module or save an empty collection]'
+    "--gitversion[Report Lmod's git version]"
     '--latest[Load latest (ignore default)]'
     '--location[Print the file location only when using show]'
     '--miniConfig[display configuration differences from default]'
     '--mt[display the module table]'
     '--no_redirect[send messages to stderr]'
     '--old_style[Use multiple module commands when mixing loads and unload]'
+    '--pin_version[pin versions when restoring a collection]'
+    '--pod[generate pod format for manpage generation]'
     '--raw[Print raw contents of a modulefile when using show]'
     '--redirect[send messages to stdout]'
     '--show_hidden[avail and spider will report hidden modules]'
     '--style[]'
     '--timer[report run times]'
+    '--terse_show_extensions[Include module extensions in terse avail output]'
     )
 
   for arg in $arguments; do

init/zsh/_module

@@ -9,27 +9,30 @@ _module()
     '(- *)'{--version,-v}'[module command version]'                                	\
     '(-D --debug)'{--debug,-D}'[Send program tracing information to stderr]'           	\
     '(-I --ignore_cache)'{--ignore_cache,-I}'[Treat the cache file(s) as out-of-date]'  \
+    '(-T --trace)'{--trace,-T}'[Report major internal actions by Lmod]'                 \
     '(-d --default)'{--default,-d}'[List default modules only when used with avail]'   	\
+    '(-f --force)'{--force,-f}'[Force removal of a sticky module or save an empty collection]' \
     '(-q --quiet)'{--quiet,-q}'[disable verbose messages]'                             	\
     '(-r --regexp)'{--regexp,-r}'[lua regular expression matching]'                     \
     '(-s --style)'{--style,-s}'[avail output style]'                                    \
     '(-t --terse)'{--terse,-t}'[display avail, list and spider output in short format]'	\
     '(-w --width)'{--width,-w}'[use given width for max term width]'                  	\
-    '--Verbose[show generated module command]'                                          \
     '--config[display full Lmod configuration]'                                         \
     '--config_json[display full Lmod configuration in json format]'                     \
-    '--force[force the removal of a sticky module or save an empty collection]'         \
+    "--gitversion[Report Lmod's git version]"                                           \
     '--latest[Load latest (ignore default)]'                                            \
     '--location[Print the file location only when using show]'                          \
     '--miniConfig[display configuration differences from default]'                      \
     '--mt[display the module table]'                                                    \
     '--no_redirect[send messages to stderr]'                                            \
-    '--old_style[Use multiple module commands when mixing loads and unload]'            \
+    '--pin_version[pin versions when restoring a collection]'                           \
+    '--pod[generate pod format for manpage generation]'                                 \
     '--raw[Print raw contents of a modulefile when using show]'                         \
     '--redirect[send messages to stdout]'                                               \
     '--show_hidden[avail and spider will report hidden modules]'                        \
     '--style[]'                                                                         \
     '--timer[report run times]'                                                         \
+    '--terse_show_extensions[Include module extensions in terse avail output]'          \
     '*::module command:_module_command'
 }
 

rt/manpath/err.txt

@@ -2,7 +2,7 @@
 step 1
 lua ProjectDIR/src/lmod.in.lua shell --regression_testing --version
 ===========================
-Modules based on Lua: Version 8.7.20  2023-03-16 11:43 -05:00
+Modules based on Lua: Version 8.7.43 2024-06-24 15:50 -06:00
     by Robert McLay [email protected]
 ===========================
 step 2
@@ -28,3 +28,11 @@ lua ProjectDIR/src/lmod.in.lua shell --regression_testing load unix
 step 7
 lua ProjectDIR/src/lmod.in.lua shell --regression_testing unload unix
 ===========================
+===========================
+step 8
+lua ProjectDIR/src/lmod.in.lua shell --regression_testing load abc
+===========================
+===========================
+step 9
+lua ProjectDIR/src/lmod.in.lua shell --regression_testing load def
+===========================

rt/manpath/manpath.tdesc

@@ -32,6 +32,10 @@ testdescript = {
      export MANPATH=/unknown/apps/junk/share/man
      runLmod load   unix                   # 6
      runLmod unload unix                   # 7
+     export MANPATH=/opt/X/man:::/opt/Y/man
+     runLmod load   abc                    # 8
+     runLmod load   def                    # 9
+
 
 
      HOME=$ORIG_HOME

rt/manpath/mf/Core/abc/1.0.lua

@@ -0,0 +1 @@
+prepend_path("MANPATH","/opt/apps/abc/share/man")

rt/manpath/mf/Core/def/2.0.lua

@@ -0,0 +1 @@
+append_path("MANPATH","/opt/apps/def//share/man")