Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add script to analyze feature generation log files #522

Merged
merged 1 commit into from
Dec 19, 2023

Conversation

bfhealy
Copy link
Collaborator

@bfhealy bfhealy commented Dec 18, 2023

This PR adds a new tools script to read slurm log files from feature generation jobs, pairing the job runtimes with the number of initial sources. When given a text file of the form "job id,runtime" from the automated slurm emails, e.g.

"25401362,16:05:28"
"25401338,11:31:10"
"25401382,1-20:20:16"
...

this code will produce a json file output that contains information relevant for predicting the runtime of future jobs, given the number of initial sources:

{"25401362": {"n_sources_start": 25190, "n_sources_end": 24443, "runtime_seconds": 57928.0, "seconds_per_source_start": 2.2996427153632393}, "25401338": {"n_sources_start": 27633, "n_sources_end": 26724, "runtime_seconds": 41470.0, "seconds_per_source_start": 1.5007418666087649}, "25401382": {"n_sources_start": 19141, "n_sources_end": 18696, "runtime_seconds": 159616.0, "seconds_per_source_start": 8.33895825714435}}

@bfhealy bfhealy temporarily deployed to Integrate Pull Request December 18, 2023 23:10 — with GitHub Actions Inactive
@bfhealy bfhealy temporarily deployed to Integrate Pull Request December 18, 2023 23:10 — with GitHub Actions Inactive
Copy link
Collaborator

@mcoughlin mcoughlin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@bfhealy bfhealy merged commit 15b4c7b into ZwickyTransientFacility:main Dec 19, 2023
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants