Filter rootnames and filenames before calling the filename_parser in archive_database_update

[bhilbert4]

The motivation for this PR is to reduce the size of the log files from archive_database_update. After #1651 the filename_parser creates a log message for files that it cannot parse. But the relevant lines are also inside a try/except. So each time a non-parsable file is used, 4-5 lines of log messages are created. In WFSS observations, there can be thousands of filenames that can't be parsed. Plus the filename_parser is actually called twice on all the files as part of archive_database_update. This was resulting in log files that were ~350MB each, once per hour.

This PR filters out those filenames before calling the filename_parser, so we should end up not having all of those logging lines added.

[pep8speaks]

Hello @bhilbert4, Thank you for updating !

Cheers ! There are no PEP8 issues in this Pull Request. 🍻

Comment last updated at 2025-01-15 17:12:29 UTC

[mfixstsci]

@bhilbert4 whipping out the re dark arts here. I think this is best way to handle this problem, thanks for submitting this.

[mfixstsci]

Oh! @bhilbert4 one last thing. I am trying to be better about this in my reviews... Would you mind submitting a test or two for this new function.

It doesn't have to be too complex

def test_filter_rootnames():
    good_rootname = ['jw_good_rootname']
    bad_rootname=['jw_bad_rootname']
    
    good_result = filter_rootnames(good_rootname)
    bad_result = filter_rootnames(bad_rootname)
    
    assert good_rootname == good_rootname  # since it should return something
    asset bad_rootname is None  # since this should return None

If you want to parameterize it or do something fancy I support that too.

[bhilbert4]

New test added! It's pretty simple, but shows that the correct level 2 files are being filtered out. Let me know if you want anything more complex.

[mfixstsci]

Nice there is a failure now, going to check on that real quick

[bhilbert4]

@mfixstsci the failures have been resolved with some shuffling of the import statements in archive_database_update