Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move copying and editing of xml files to 'runTestsAndObjectiveCalc.py' #6

Open
cpecar opened this issue Feb 12, 2024 · 2 comments
Open

Comments

@cpecar
Copy link
Collaborator

cpecar commented Feb 12, 2024

Currently, new xml files are created for a design point to submitting the slurm job which runs the objective calculation/overlap check for the given point slurm wrapper . This can be better parallelized if the xml files are created within the slurm job.

@cpecar
Copy link
Collaborator Author

cpecar commented Feb 14, 2024

@karthik18495 after looking back into this, I think creating the xml files from the SlurmQueueClient makes sense at least for the slurm implementation, as otherwise the dict of the design parameters would have to somehow be passed through the master slurm job for that trial, as I'm picturing this as:

  1. From SlurmQueueClient create xml files and launch master slurm job for the trial (calls a python script that knows which xml files to load)
  2. In master slurm job, launch a job simulating each p-eta bin, and wait for them to complete.
  3. After all sub-jobs complete, compute desired objectives

However, I agree that it should be moved elsewhere for the joblib implementation.

@karthik18495
Copy link
Member

Right sure. I agree with these steps. It would be great to also keep thinking on having a common software design across all these runners. We are going to have 3 runners

  1. SlurmRunner
  2. JoblibRunner
  3. PanDARunner

I shall by tomorrow create a new pull request for JoblibRunner.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants