rewrite opendap_post.sh

[wwlwpd]

Major clean up and rewrite

[wwlwpd]

A major thrust of this work will be the ability to easily, manually "rerun" this for any run. This will allow for a quick attempt to retry issues during a storm; but also for more rapid testing/development.

[jasonfleming]

This should also include refinement of the list of files being posted in light of #607 including redacting sensitive info like user names, email addresses, asgs config files, etc

[wwlwpd]

This should also include refinement of the list of files being posted in light of #607 including redacting sensitive info like user names, email addresses, asgs config files, etc

10-4

[jasonfleming]

There are many features built into opendap_post.sh.

Issues with Status Quo

• Supporting more than one xfer method in one file
• Building in failover in the loop itself
• Remote commands are inline

Architecture Overview
The following should be performed as separate steps/scripts:

1. Create list of files to send
2. Send files (scp)
3. Notify (email)

Feature Requirements

1. Automated execution from ASGS as post processing script
2. Manual execution by Operator to re-post files
3. Ad hoc execution where Operator can specify a list of files and a destination and execute the script
4. Support posting files via scp, rsync, as well as cp, and ln -s (just scp to start) with different scripts to implement different methods
5. Ssh used to set dir structure on remote machine
6. Need to modify permissions on remote machine in an error-free way so that other Operators can post files into the same directory hierarchy
7. Maintain portability across HPC platforms, THREDDS server platforms, and all realistic combinations of those
8. Bulletproof reliability (checking exit codes, implementing retries etc)
9. Implement symbolic link from storm name to basin stormnumber year (e.g. al092021 -> Ida) ... this doesn't work at TACC (corral vs vm ... different paths)
10. Also want to post status files like hook.status.json and scenario.status.json to correct paths
11. Support posting input files to separate archive (e.g., BackBlaze)
12. Record ping time to log file as well as ping failure to visible place
13. Redact sensitive information before posting
14. Make all parameters reconfigurable on the fly without restarting ASGS
15. Resolve all the ASGS issues tagged opendap

Techniques

• Everything comes from run.properties
• Also consider having asgs write a json file to control execution of opendap_post2.sh
• A template approach is not out of the question
  • Perl script writes a shell script on the fly that asgs executes to post the files
  • For template approach

POSTPROCESS=( opendap_post2.sh )

in opendap_post2.sh

execute perl script to generate json
execute perl script (local or remote) to create bash script that would post the files (request/response)
execute bash script just created
exit

Alt Techniques

• Centralized orchestration
  • HPCs as computational environment
  • Coordination of input and output files is managed elsewhere
  • Handled this way at NRL with securerun
  • Write xml file, use ssh to interact with the machine, submit the job, etc
• Remote Orchestration
  • Rather than have service to generate script and drop locally
  • Could be an api call to send essential info
  • Uploads files to service that knows where files go and cxn info
  • ASGS does not need to know how to get to thredds server
  • Acts as transfer middleware
  • Centralized place that manages distribution of files ;
  • Pull vs push to Corral at TACC?

[wwlwpd]

output/ssh_only.sh has been added as of #811; so this issue should take that into account.