Conference Notes for 2021 10 05

• Debugging Modulefiles Discussion
• Issue #531: depends_on doesn't warn when swapping versions
• Ward Poelmans: Lmod configuration file?
• Better shell script handling
• Slide Presentation: Internals of Lmod: How Lmod handles loading/unloading/spidering modulefiles

Attendees

• Robert McLay (TACC, US)
• Kenneth Hoste (HPC-UGent, Belgium)
• Kurt Lust (Univ. of Antwerp (Belgium) + LUMI)
• Shelly Johnson (Univ. of Michigan, US)
• Bennet Fauber (Univ. of Michigan, US)
• Mikael Öhman (Chalmers University of Technology, Sweden)

Notes

Debugging module files

• enable tracing (debugging "light"): ml -T to see what's happening during the load
• see what changes to environment Lmod would do via $LMOD_CMD bash load
• ml -mt shows module table that is stored in environment
  • is stored in blocks in $_ModuleTableXYZ_ environment variables
• debug output when loading a module via ml -D
  • often asked when trying to debug reported problems: ml -D hdf5 &> lmod.log

depends_on doesn't warn when swapping versions

• see https://github.com/TACC/Lmod/issues/531
• no warnings from depends_on when loading modules that depend on different compiler versions
• Mikael solemnly sweared to come up with a minimal example to reproduce this issue

Lmod configuration file

• for example /etc/lmod.cfg
• separate way of configuring Lmod, next to build-time options and $LMOD_* environment variables
• Robert will sync up with Ward on this
• Kenneth/Bennet/Kurt agree that this would be a useful feature
  • Bennet: single place to look at Lmod configuration, easily shareable
  • Kurt: also useful when using an Lmod built by vendor (e.g. on Cray)

Better shell script handling

• Tmod 4.6.x can source shell script from module files
  • generates a script that dumps environment before sourcing script
  • can restore changes made by sourced script when module is unloaded
• Robert is actively working on this
• in theory it's possible to write a shell script that does stuff that can't be reversed easily
  • but "normal" changes to the environment should be reversible
• goal is to be able to handle known script like:
  • OpenFOAM's $FOAM_BASH
  • Intel script to set up environment for compilers
  • conda activate

Internals of Lmod: How Lmod handles loading/unloading/spidering modulefiles

• see https://github.com/TACC/Lmod/blob/master/my_docs/21/lmod_zoom_mtg_2021_10_05/presentation.pdf
• multiple ways to evaluate module files
  • 3 main ways: load, unload, show
  • 10 ways in total
    • like refresh (redo setting of shell functions and aliases, useful in subshell)
    • help ignores everything, except the help function
  • actions specified in module file are reversed when unloading a module file
    • changes made by loading are not stored in environment (only end result is)
  • in Lmod codebase:
    • MasterControl object + deriving classes for Load, Unload, etc.
    • varT contains full set of environment variables + values
    • MCP (caps) is always load action, mcp (lowercase) is currently active "mode" in which module file is processed
    • implementations of support for statements like setenv leverage logic in MasterControl.lua that knows what to do depending on the "mode" in which a module file is being processed
  • which other internal Lmod topics would be worth discussing?
    • (see slide 16 in presentation)
    • MName object represents a module name, but there's more to it than just the pure module name
    • code style: variable can end in T (hash table, key value pairs), A for variables used as arrays
    • Kenneth: explain through which functions you travel when running a module command
    • Kenneth: birds-eye view of codebase, how Lmod codebase is organised?
      • something for the Lmod docs?
    • Bennet: more on how module collections work
      • some problems with restoring module collections from system A on system B with slightly different setup
      • Robert: can fix this by letting Lmod know these are different systems (with a shared home filesystem)
        • see docs
• questions:
  • why is restore only "kinda"?
  • pushenv alternative to setenv (to restore original value on unload)

Future topics

• Bennet: settarg
  • cool topic to cover, mainly focused on developers

Next meeting

• 2nd Tuesday of Nov (to avoid timezone mixup on 1st Tuesday)