Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Release v8.0.0 #2463

Merged
merged 193 commits into from
Oct 24, 2024
Merged
Changes from 1 commit
Commits
Show all changes
193 commits
Select commit Hold shift + click to select a range
360da7d
rough draft of SRG_TIME_SERIES.yml
jtherrmann Oct 2, 2024
49ad39d
items: granules
jtherrmann Oct 2, 2024
9a7cf51
rename steps to states
jtherrmann Oct 2, 2024
fd9fc5e
rename back to steps
jtherrmann Oct 2, 2024
f127a66
eliminate map nesting
jtherrmann Oct 2, 2024
a6f991d
simplify tasks
jtherrmann Oct 2, 2024
d31a620
construct step function json for jobs in render.py
jtherrmann Oct 3, 2024
6ff153f
render map state for step function
jtherrmann Oct 4, 2024
3cd0026
max 300 granules
jtherrmann Oct 4, 2024
c217c33
do not specify default params in Batch job definition
jtherrmann Oct 4, 2024
0dbe157
rename SRG_TIME_SERIES custom compute env
jtherrmann Oct 4, 2024
2aa1699
update hyp3-multi-burst-sandbox to match hyp3-lavas-test
jtherrmann Oct 4, 2024
603a487
set placeholder timeout and memory for srg time series
jtherrmann Oct 4, 2024
8eae234
Deploy multi-burst-sandbox from srg branch, fix sfn name collision
jtherrmann Oct 4, 2024
33e758a
fix sfn step name, use hyp3-srg dev image
jtherrmann Oct 4, 2024
4088b18
pass original array to Map state rather than stringified version
jtherrmann Oct 4, 2024
da11d26
add back job definition default params, fix image tags
jtherrmann Oct 4, 2024
6b99bbf
pass bucket prefix and other params to SRG jobs
jtherrmann Oct 4, 2024
f0c1a24
update image tag
jtherrmann Oct 4, 2024
2622ebe
add granules to bucket prefix in job spec
jtherrmann Oct 4, 2024
5281585
do not add granules to bucket prefix in job spec
jtherrmann Oct 4, 2024
e0000a4
run time series back projection step in default compute env for now
jtherrmann Oct 4, 2024
3d7337f
pass --use-granules-from-s3 to time_series workflow
jtherrmann Oct 5, 2024
bd8f4a5
image tag
jtherrmann Oct 5, 2024
b27b909
increase timeout for time series gslc step
jtherrmann Oct 5, 2024
8021bcd
image tag
jtherrmann Oct 5, 2024
f3c9ed1
image tag
jtherrmann Oct 7, 2024
fac1172
per task compute_envs
AndrewPlayer3 Oct 7, 2024
782a8c7
add `do` extension
AndrewPlayer3 Oct 7, 2024
50d0e6c
move compute env into tasks
AndrewPlayer3 Oct 7, 2024
7820251
moving SRG compute env to file
AndrewPlayer3 Oct 7, 2024
b0d52e7
Merge branch 'srg' of https://github.com/asfhyp3/hyp3 into per-task-c…
AndrewPlayer3 Oct 7, 2024
6163404
added from_file compute env capability
AndrewPlayer3 Oct 7, 2024
21aca1b
removed -vv for tests
AndrewPlayer3 Oct 7, 2024
3dfb367
dont dupe names
AndrewPlayer3 Oct 7, 2024
721394e
dont dupe envs
AndrewPlayer3 Oct 7, 2024
4a46dbf
Merge branch 'develop' into srg
jtherrmann Oct 7, 2024
a7b6c26
Merge branch 'srg' into per-task-compute-envs
jtherrmann Oct 7, 2024
5d80472
changed 'from_file' to 'import'
AndrewPlayer3 Oct 8, 2024
880dfb6
Merge branch 'per-task-compute-envs' of https://github.com/asfhyp3/hy…
AndrewPlayer3 Oct 8, 2024
bb2aad1
rename compute_environments folder
AndrewPlayer3 Oct 8, 2024
55504ff
insariscearia compute env
AndrewPlayer3 Oct 8, 2024
80a0f49
update check_processing_time to handle list of results
jtherrmann Oct 8, 2024
9ce5fdc
moved compute env logic to render_cf.py
AndrewPlayer3 Oct 8, 2024
d3d5702
refactor
AndrewPlayer3 Oct 9, 2024
fe248e5
Merge branch 'srg' into per-task-compute-envs
jtherrmann Oct 9, 2024
95c63eb
add exception to imports
AndrewPlayer3 Oct 9, 2024
f7657e1
Merge branch 'per-task-compute-envs' of https://github.com/asfhyp3/hy…
AndrewPlayer3 Oct 9, 2024
3b39f14
Update apps/render_cf.py
AndrewPlayer3 Oct 9, 2024
bc96682
compute_env_name to name
AndrewPlayer3 Oct 9, 2024
a08debc
Merge branch 'per-task-compute-envs' of https://github.com/asfhyp3/hy…
AndrewPlayer3 Oct 9, 2024
456561b
only parse compute env file if one was given
AndrewPlayer3 Oct 9, 2024
fd748b1
ensure that imported compute envs are defined
AndrewPlayer3 Oct 9, 2024
6bbb1de
Update apps/render_cf.py
AndrewPlayer3 Oct 10, 2024
9ae8fb1
Update apps/render_cf.py
AndrewPlayer3 Oct 10, 2024
76e0fc4
Update apps/render_cf.py
AndrewPlayer3 Oct 10, 2024
146769c
Update Makefile
AndrewPlayer3 Oct 10, 2024
071ce78
Update Makefile
AndrewPlayer3 Oct 10, 2024
692a8ef
Update apps/compute-cf.yml.j2
AndrewPlayer3 Oct 10, 2024
802e8b4
Update apps/render_cf.py
AndrewPlayer3 Oct 10, 2024
2a2ce44
Update apps/render_cf.py
AndrewPlayer3 Oct 10, 2024
78b8dc6
add Optional and change list to set
AndrewPlayer3 Oct 10, 2024
3f42c31
Merge branch 'develop' into srg
jtherrmann Oct 10, 2024
ae71a16
Update apps/render_cf.py
AndrewPlayer3 Oct 10, 2024
7956857
Update apps/render_cf.py
AndrewPlayer3 Oct 10, 2024
8c485e7
Merge pull request #2453 from ASFHyP3/per-task-compute-envs
jtherrmann Oct 10, 2024
c4f14fb
update srg time series job spec
jtherrmann Oct 10, 2024
6d832e4
do not overwrite original job params in start execution worker
jtherrmann Oct 10, 2024
4362d49
rename job tasks to job steps
jtherrmann Oct 11, 2024
6e1f4d6
add validators
AndrewPlayer3 Oct 11, 2024
0f585d9
add validators to job_spec
AndrewPlayer3 Oct 11, 2024
b7dcfbf
add validators to job_spec
AndrewPlayer3 Oct 11, 2024
227dc50
flake8
AndrewPlayer3 Oct 11, 2024
c84e346
Merge branch 'srg' of https://github.com/asfhyp3/hyp3 into srg-valida…
AndrewPlayer3 Oct 11, 2024
b6a7094
better bounds description
AndrewPlayer3 Oct 11, 2024
ee21a5d
Update job_spec/SRG_GSLC.yml
AndrewPlayer3 Oct 11, 2024
5515e47
Update job_spec/SRG_TIME_SERIES.yml
AndrewPlayer3 Oct 11, 2024
48045b8
Update apps/api/src/hyp3_api/validation.py
AndrewPlayer3 Oct 11, 2024
1c40c13
Update apps/api/src/hyp3_api/validation.py
AndrewPlayer3 Oct 11, 2024
aae02ac
Update apps/api/src/hyp3_api/validation.py
AndrewPlayer3 Oct 11, 2024
0a572b7
handle default bounds in check_granules_intersecting_bounds
AndrewPlayer3 Oct 11, 2024
ec9929a
handle default bounds case
AndrewPlayer3 Oct 11, 2024
7d21eb2
extra test cases
AndrewPlayer3 Oct 11, 2024
0964444
use gpu image for gslc map step
jtherrmann Oct 11, 2024
6d486a6
Update tests/test_api/test_validation.py
jtherrmann Oct 11, 2024
8d3c7c7
Merge pull request #2455 from ASFHyP3/srg-validators
jtherrmann Oct 11, 2024
2212e4a
revert image tag for time_series step back to CPU version
jtherrmann Oct 11, 2024
dd94f5d
image tags
jtherrmann Oct 11, 2024
3fe2592
image tags
jtherrmann Oct 14, 2024
77b3272
Bump cfn-lint from 1.16.0 to 1.16.1
dependabot[bot] Oct 14, 2024
e557088
Bump boto3 from 1.35.34 to 1.35.40
dependabot[bot] Oct 14, 2024
9413085
image tag
jtherrmann Oct 14, 2024
d65e906
remove old todo
jtherrmann Oct 14, 2024
dea1718
image tags
jtherrmann Oct 14, 2024
f05c6c9
increase srg time series max to 600, fix default example
jtherrmann Oct 14, 2024
6b6da4a
update processing_times api spec
jtherrmann Oct 14, 2024
11704f9
fix proc time api schema
jtherrmann Oct 14, 2024
be3d81b
add relative orbit check for srg time series
AndrewPlayer3 Oct 15, 2024
4e4d024
remove else
AndrewPlayer3 Oct 15, 2024
c02c4eb
refactor
AndrewPlayer3 Oct 15, 2024
54d898f
list to number
AndrewPlayer3 Oct 15, 2024
5c56489
flake8
AndrewPlayer3 Oct 15, 2024
1ef5d15
refactor
AndrewPlayer3 Oct 15, 2024
1684724
add relative orbit check to srg_gslc
AndrewPlayer3 Oct 15, 2024
4666bea
add reference for relative orbit calc
AndrewPlayer3 Oct 15, 2024
51fea7b
image tags
jtherrmann Oct 15, 2024
d507b45
filter batch job output in step function
jtherrmann Oct 15, 2024
d9eadf0
json path *
jtherrmann Oct 15, 2024
15c7813
refactor
AndrewPlayer3 Oct 16, 2024
61cf1d3
allow excluding job params
jtherrmann Oct 16, 2024
7585fed
fix typo
jtherrmann Oct 16, 2024
0ba47e3
fix step function batch params key
jtherrmann Oct 16, 2024
11de54a
Merge pull request #2459 from ASFHyP3/srg-validators
AndrewPlayer3 Oct 16, 2024
865308b
relative orbits unit test is failing
jtherrmann Oct 16, 2024
83a374a
automatically filter batch params in step function
jtherrmann Oct 16, 2024
e5a731d
readability
jtherrmann Oct 16, 2024
5de24c8
filter stringified batch params in start_execution_worker
jtherrmann Oct 16, 2024
dfb93fa
pass entire job
jtherrmann Oct 16, 2024
ef1efd1
gitignore
jtherrmann Oct 16, 2024
dd83213
Merge branch 'develop' into srg
jtherrmann Oct 16, 2024
88f2f06
test fix
AndrewPlayer3 Oct 17, 2024
5046575
Merge pull request #2462 from ASFHyP3/srg-validators
jtherrmann Oct 17, 2024
94eb5ec
remove job definition default param values
jtherrmann Oct 17, 2024
8e54a2a
try removing non-api-schema default job params
jtherrmann Oct 17, 2024
0b171b5
fix SRG_GSLC
jtherrmann Oct 17, 2024
06422a2
include bucket_prefix in batch params
jtherrmann Oct 17, 2024
7e2502e
assume all params have api_schema in api spec template
jtherrmann Oct 17, 2024
98727e6
finish removing non-api-schema defaults from job specs
jtherrmann Oct 17, 2024
f919149
image tag
jtherrmann Oct 17, 2024
6cdec5e
fix check_processing_time unit tests
jtherrmann Oct 17, 2024
01e4f43
fix `test_start_execution_worker.py::test_submit_jobs`
jtherrmann Oct 17, 2024
e19abb7
move a comment, delete old unit test
jtherrmann Oct 17, 2024
335457c
image tags
jtherrmann Oct 17, 2024
bd25c2b
image tags
jtherrmann Oct 18, 2024
aaa8f5c
validate job specs
jtherrmann Oct 18, 2024
2be9e83
job_step -> step
jtherrmann Oct 18, 2024
0bbbadc
time series todos, start changelog
jtherrmann Oct 18, 2024
573d956
add new srg job to lavas deployments, finish changelog
jtherrmann Oct 18, 2024
6fc0908
tweaks
jtherrmann Oct 18, 2024
5ac84a6
wording
jtherrmann Oct 18, 2024
1b8532a
flake8
jtherrmann Oct 18, 2024
2b596c5
Update job_spec/INSAR_ISCE.yml
jtherrmann Oct 18, 2024
9ec6676
Update image tags
jtherrmann Oct 18, 2024
5f10cce
Update job_spec/SRG_TIME_SERIES.yml
jtherrmann Oct 18, 2024
24856d5
Update apps/render_cf.py
jtherrmann Oct 18, 2024
b932fdc
Merge pull request #2447 from ASFHyP3/srg
jtherrmann Oct 18, 2024
80f6c0d
Merge pull request #2458 from ASFHyP3/dependabot/pip/boto3-1.35.40
jtherrmann Oct 18, 2024
a90e1c7
Merge pull request #2457 from ASFHyP3/dependabot/pip/cfn-lint-1.16.1
jtherrmann Oct 18, 2024
ab5f6af
Bump moto[dynamodb] from 5.0.16 to 5.0.17
dependabot[bot] Oct 18, 2024
20fb65d
Merge pull request #2456 from ASFHyP3/dependabot/pip/moto-dynamodb--5…
jtherrmann Oct 18, 2024
24a7703
Remove .cpu and .gpu image tags from SRG job_specs
AndrewPlayer3 Oct 18, 2024
a77eddb
Merge pull request #2464 from ASFHyP3/remove-srg-image-tags
AndrewPlayer3 Oct 18, 2024
c6dc842
refactor compute env parsing so that only used compute envs are rendered
AndrewPlayer3 Oct 18, 2024
fc926ec
Merge branch 'develop' of https://github.com/asfhyp3/hyp3 into refact…
AndrewPlayer3 Oct 18, 2024
7d0ade1
added all important documentation
AndrewPlayer3 Oct 18, 2024
5123669
re-ordered comment
AndrewPlayer3 Oct 18, 2024
4a8e490
add test for get_compute_environments
AndrewPlayer3 Oct 21, 2024
1852c36
error when user imports but doesnt provide file
AndrewPlayer3 Oct 21, 2024
2ac2480
add error cases to test_get_compute_environments
AndrewPlayer3 Oct 21, 2024
ddc4c10
flake8
AndrewPlayer3 Oct 21, 2024
c5fbbe4
flake8
AndrewPlayer3 Oct 21, 2024
ec0d26c
Update Makefile
AndrewPlayer3 Oct 21, 2024
3062516
Update Makefile
AndrewPlayer3 Oct 21, 2024
4538d57
Update apps/render_cf.py
AndrewPlayer3 Oct 21, 2024
dcad83b
Bump cryptography from 43.0.1 to 43.0.3
dependabot[bot] Oct 21, 2024
2dabafb
Bump setuptools from 75.1.0 to 75.2.0
dependabot[bot] Oct 21, 2024
0b95b53
Bump boto3 from 1.35.40 to 1.35.44
dependabot[bot] Oct 21, 2024
b0c8e43
Bump ASFHyP3/actions from 0.11.2 to 0.12.0
dependabot[bot] Oct 21, 2024
4c4802e
test parse_job_step_map
jtherrmann Oct 21, 2024
2f0b387
improve map statement parser error handling
jtherrmann Oct 21, 2024
08c72ba
placeholder tests
jtherrmann Oct 21, 2024
2161a86
restrict custom compute env defs to one file
jtherrmann Oct 21, 2024
e258b70
combine test files
jtherrmann Oct 21, 2024
7c0557e
update a test
jtherrmann Oct 22, 2024
25e91e8
implement test
jtherrmann Oct 22, 2024
25d9b7e
implement a test and improve function
jtherrmann Oct 22, 2024
7f86585
adjust newline
jtherrmann Oct 22, 2024
f000d98
Merge pull request #2472 from ASFHyP3/jth-render-tests
AndrewPlayer3 Oct 22, 2024
631e8a6
update changelog
jtherrmann Oct 22, 2024
d3afd83
Merge pull request #2465 from ASFHyP3/refactor-compute-envs
jtherrmann Oct 22, 2024
d2cb0ae
revert SRG changes to multi-burst-sandbox
jtherrmann Oct 22, 2024
da772bc
Merge pull request #2473 from ASFHyP3/multi-burst-sandbox
jtherrmann Oct 22, 2024
09f6acd
add boundsvalidationerror
AndrewPlayer3 Oct 22, 2024
6fb1c8d
remove do extension
AndrewPlayer3 Oct 22, 2024
7321873
Merge pull request #2474 from ASFHyP3/bounds-error-fix
AndrewPlayer3 Oct 22, 2024
8670bad
Merge pull request #2471 from ASFHyP3/dependabot/github_actions/ASFHy…
jtherrmann Oct 23, 2024
43135b2
Merge pull request #2469 from ASFHyP3/dependabot/pip/boto3-1.35.44
jtherrmann Oct 23, 2024
0d96051
Merge pull request #2468 from ASFHyP3/dependabot/pip/setuptools-75.2.0
jtherrmann Oct 23, 2024
b49b4e1
Bump cfn-lint from 1.16.1 to 1.18.1
dependabot[bot] Oct 23, 2024
19471a8
Bump moto[dynamodb] from 5.0.17 to 5.0.18
dependabot[bot] Oct 23, 2024
ee202a2
Merge pull request #2475 from ASFHyP3/dependabot/pip/cfn-lint-1.18.1
jtherrmann Oct 23, 2024
7789788
Merge pull request #2466 from ASFHyP3/dependabot/pip/cryptography-43.0.3
jtherrmann Oct 24, 2024
35a11ca
Merge pull request #2470 from ASFHyP3/dependabot/pip/moto-dynamodb--5…
jtherrmann Oct 24, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
do not overwrite original job params in start execution worker
  • Loading branch information
jtherrmann committed Oct 10, 2024

Verified

This commit was created on GitHub.com and signed with GitHub’s verified signature. The key has expired.
commit 6d832e4110f16d341f341dcf9680c930ece31c49
18 changes: 9 additions & 9 deletions apps/render_cf.py
Original file line number Diff line number Diff line change
@@ -54,18 +54,18 @@ def get_step_for_task(task: dict, index: int, next_step_name: str, job_spec: dic

def get_step_for_map_task(task: dict, job_spec: dict) -> dict:
item, items = parse_task_map(task['map'])
job_parameters = get_job_parameters(item, items, job_spec)
batch_job_parameters = get_batch_job_parameters(item, items, job_spec)
submit_job_step = get_step_for_batch_submit_job(task)
submit_job_step['End'] = True
submit_job_step_name = task['name'] + '_SUBMIT_JOB'
return {
'Type': 'Map',
'ItemsPath': f'$.original_job_parameters.{items}',
'ItemsPath': f'$.job_parameters.{items}',
'ItemSelector': {
'job_id.$': '$.job_id',
'priority.$': '$.priority',
'container_overrides.$': '$.container_overrides',
'job_parameters': job_parameters,
'batch_job_parameters': batch_job_parameters,
},
'ItemProcessor': {
'StartAt': submit_job_step_name,
@@ -83,14 +83,14 @@ def parse_task_map(task_map: str) -> tuple[str, str]:
return tokens[1], tokens[3]


def get_job_parameters(item: str, items: str, job_spec: dict) -> dict:
job_parameters = {
f'{param}.$': f'$.job_parameters.{param}'
def get_batch_job_parameters(item: str, items: str, job_spec: dict) -> dict:
batch_job_parameters = {
f'{param}.$': f'$.batch_job_parameters.{param}'
for param in job_spec['parameters']
if param != items
}
job_parameters[f'{item}.$'] = '$$.Map.Item.Value'
return job_parameters
batch_job_parameters[f'{item}.$'] = '$$.Map.Item.Value'
return batch_job_parameters


def get_step_for_batch_submit_job(task: dict) -> dict:
@@ -108,7 +108,7 @@ def get_step_for_batch_submit_job(task: dict) -> dict:
'JobQueue': '${' + job_queue + '}',
'ShareIdentifier': 'default',
'SchedulingPriorityOverride.$': '$.priority',
'Parameters.$': '$.job_parameters',
'Parameters.$': '$.batch_job_parameters',
'ContainerOverrides.$': '$.container_overrides',
'RetryStrategy': {
'Attempts': 3
10 changes: 5 additions & 5 deletions apps/set-batch-overrides/src/set_batch_overrides.py
Original file line number Diff line number Diff line change
@@ -35,7 +35,7 @@ def get_container_overrides(memory: str, omp_num_threads: str = None) -> dict:

def get_insar_isce_burst_memory(job_parameters: dict) -> str:
looks = job_parameters['looks']
bursts = len(job_parameters['reference'].split(' '))
bursts = len(job_parameters['reference'])
if looks == '5x1':
if bursts < 2:
return INSAR_ISCE_BURST_MEMORY_8G
@@ -70,16 +70,16 @@ def lambda_handler(event: dict, _) -> dict:
omp_num_threads = INSAR_ISCE_BURST_OMP_NUM_THREADS[memory]
return get_container_overrides(memory, omp_num_threads)

if job_type == 'AUTORIFT' and job_parameters['granules'].startswith('S2'):
if job_type == 'AUTORIFT' and job_parameters['granules'][0].startswith('S2'):
return get_container_overrides(AUTORIFT_S2_MEMORY)

if job_type == 'AUTORIFT' and job_parameters['granules'].startswith('L'):
if job_type == 'AUTORIFT' and job_parameters['granules'][0].startswith('L'):
return get_container_overrides(AUTORIFT_LANDSAT_MEMORY)

if job_type == 'RTC_GAMMA' and job_parameters['resolution'] in ['10', '20']:
if job_type == 'RTC_GAMMA' and job_parameters['resolution'] in [10, 20]:
return get_container_overrides(RTC_GAMMA_10M_MEMORY)

if job_type in ['WATER_MAP', 'WATER_MAP_EQ'] and job_parameters['resolution'] in ['10', '20']:
if job_type in ['WATER_MAP', 'WATER_MAP_EQ'] and job_parameters['resolution'] in [10, 20]:
return get_container_overrides(WATER_MAP_10M_MEMORY)

return {}
7 changes: 1 addition & 6 deletions apps/start-execution-worker/src/start_execution_worker.py
Original file line number Diff line number Diff line change
@@ -25,12 +25,7 @@ def submit_jobs(jobs: list[dict]) -> None:
for job in jobs:
# Convert parameters to strings so they can be passed to Batch; see:
# https://docs.aws.amazon.com/batch/latest/APIReference/API_SubmitJob.html#Batch-SubmitJob-request-parameters

# TODO: instead, assign the stringified params to a batch_job_parameters key, and keep the original job_parameters
# value unchanged; will require searching for usage of job_parameters in step-function.json.j2 and render_cf.py
# and updating how they're used
job['original_job_parameters'] = job['job_parameters']
job['job_parameters'] = convert_parameters_to_strings(job['job_parameters'])
job['batch_job_parameters'] = convert_parameters_to_strings(job['job_parameters'])
STEP_FUNCTION.start_execution(
stateMachineArn=step_function_arn,
input=json.dumps(job, sort_keys=True),
2 changes: 1 addition & 1 deletion apps/step-function.json.j2
Original file line number Diff line number Diff line change
@@ -35,7 +35,7 @@
"ADD_PREFIX_TO_JOB_PARAMETERS": {
"Type": "Pass",
"InputPath": "$.job_id",
"ResultPath": "$.job_parameters.bucket_prefix",
"ResultPath": "$.batch_job_parameters.bucket_prefix",
"Next": "SET_BATCH_OVERRIDES"
},
"SET_BATCH_OVERRIDES": {
18 changes: 9 additions & 9 deletions tests/test_set_batch_overrides.py
Original file line number Diff line number Diff line change
@@ -19,7 +19,7 @@ def mock_insar_isce_burst_job(looks: str, bursts: int) -> dict:
'job_type': 'INSAR_ISCE_MULTI_BURST',
'job_parameters': {
'looks': looks,
'reference': ' '.join('foo' for _ in range(bursts)),
'reference': ['foo' for _ in range(bursts)],
}
}

@@ -154,7 +154,7 @@ def test_set_batch_overrides_autorift_s2():
assert lambda_handler(
{
'job_type': 'AUTORIFT',
'job_parameters': {'granules': 'S2B_'},
'job_parameters': {'granules': ['S2B_']},
},
None,
) == {
@@ -171,7 +171,7 @@ def test_set_batch_overrides_autorift_landsat():
assert lambda_handler(
{
'job_type': 'AUTORIFT',
'job_parameters': {'granules': 'LC08_'},
'job_parameters': {'granules': ['LC08_']},
},
None,
) == {
@@ -188,7 +188,7 @@ def test_set_batch_overrides_rtc_gamma_10m():
assert lambda_handler(
{
'job_type': 'RTC_GAMMA',
'job_parameters': {'resolution': '10'},
'job_parameters': {'resolution': 10},
},
None,
) == {
@@ -202,7 +202,7 @@ def test_set_batch_overrides_rtc_gamma_10m():
assert lambda_handler(
{
'job_type': 'RTC_GAMMA',
'job_parameters': {'resolution': '20'},
'job_parameters': {'resolution': 20},
},
None,
) == {
@@ -219,7 +219,7 @@ def test_set_batch_overrides_water_map_10m():
assert lambda_handler(
{
'job_type': 'WATER_MAP',
'job_parameters': {'resolution': '10'},
'job_parameters': {'resolution': 10},
},
None,
) == {
@@ -233,7 +233,7 @@ def test_set_batch_overrides_water_map_10m():
assert lambda_handler(
{
'job_type': 'WATER_MAP',
'job_parameters': {'resolution': '20'},
'job_parameters': {'resolution': 20},
},
None,
) == {
@@ -247,7 +247,7 @@ def test_set_batch_overrides_water_map_10m():
assert lambda_handler(
{
'job_type': 'WATER_MAP_EQ',
'job_parameters': {'resolution': '10'},
'job_parameters': {'resolution': 10},
},
None,
) == {
@@ -261,7 +261,7 @@ def test_set_batch_overrides_water_map_10m():
assert lambda_handler(
{
'job_type': 'WATER_MAP_EQ',
'job_parameters': {'resolution': '20'},
'job_parameters': {'resolution': 20},
},
None,
) == {
13 changes: 12 additions & 1 deletion tests/test_start_execution_worker.py
Original file line number Diff line number Diff line change
@@ -63,6 +63,16 @@ def test_submit_jobs():
'float_field': 10.1,
'integer_field': 10,
'job_parameters': {
'granules': [
'granule1',
'granule2',
],
'string_field': 'value1',
'boolean_field': True,
'float_field': 10.1,
'integer_field': 10,
},
'batch_job_parameters': {
'granules': 'granule1 granule2',
'string_field': 'value1',
'boolean_field': 'True',
@@ -76,7 +86,8 @@ def test_submit_jobs():
expected_input_job1 = json.dumps(
{
'job_id': 'job1',
'job_parameters': {'granules': 'granule1'},
'job_parameters': {'granules': ['granule1']},
'batch_job_parameters': {'granules': 'granule1'},
},
sort_keys=True,
)