Gabriela Palacios
01/05/2023, 4:27 AM/home/ubuntu/.pyenv/versions/env_prefect2/lib/python3.8/site-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated
"class": algorithms.Blowfish,
21:02:25.829 | INFO | prefect.engine - Created flow run 'merciful-bee' for flow 'flow-hawaii'
21:02:26.659 | INFO | Flow run 'merciful-bee' - Created task run 'schema_popup-eff23439-0' for task 'schema_popup'
21:02:26.660 | INFO | Flow run 'merciful-bee' - Executing 'schema_popup-eff23439-0' immediately...
21:02:27.266 | INFO | Task run 'schema_popup-eff23439-0' - Finished in state Completed()
21:02:27.395 | INFO | Flow run 'merciful-bee' - Created task run 'check_if_historical-eaf1f601-0' for task 'check_if_historical'
21:02:27.395 | INFO | Flow run 'merciful-bee' - Executing 'check_if_historical-eaf1f601-0' immediately...
21:02:27.712 | INFO | Task run 'check_if_historical-eaf1f601-0' - Finished in state Completed()
is historical is False
AWS credentials are AwsCredentials(aws_access_key_id=None, aws_secret_access_key=None, aws_session_token=None, profile_name=None, region_name=None)
21:02:27.799 | INFO | Flow run 'merciful-bee' - Created task run 'client_waiter-c67b07b1-0' for task 'client_waiter'
21:02:27.799 | INFO | Flow run 'merciful-bee' - Executing 'client_waiter-c67b07b1-0' immediately...
21:02:28.021 | INFO | Task run 'client_waiter-c67b07b1-0' - Waiting on batch job
21:02:28.065 | ERROR | Task run 'client_waiter-c67b07b1-0' - Encountered exception during execution:
Traceback (most recent call last):
File "/home/ubuntu/.pyenv/versions/env_prefect2/lib/python3.8/site-packages/prefect/engine.py", line 1215, in orchestrate_task_run
result = await task.fn(*args, **kwargs)
File "/home/ubuntu/.pyenv/versions/env_prefect2/lib/python3.8/site-packages/prefect_aws/client_waiter.py", line 76, in client_waiter
await run_sync_in_worker_thread(waiter.wait, **waiter_kwargs)
File "/home/ubuntu/.pyenv/versions/env_prefect2/lib/python3.8/site-packages/prefect/utilities/asyncutils.py", line 68, in run_sync_in_worker_thread
return await anyio.to_thread.run_sync(call, cancellable=True)
File "/home/ubuntu/.pyenv/versions/env_prefect2/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File "/home/ubuntu/.pyenv/versions/env_prefect2/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
return await future
File "/home/ubuntu/.pyenv/versions/env_prefect2/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
result = context.run(func, *args)
File "/home/ubuntu/.pyenv/versions/env_prefect2/lib/python3.8/site-packages/botocore/waiter.py", line 55, in wait
Waiter.wait(self, **kwargs)
File "/home/ubuntu/.pyenv/versions/env_prefect2/lib/python3.8/site-packages/botocore/waiter.py", line 343, in wait
response = self._operation_method(**kwargs)
File "/home/ubuntu/.pyenv/versions/env_prefect2/lib/python3.8/site-packages/botocore/waiter.py", line 93, in _call_
return self._client_method(**kwargs)
File "/home/ubuntu/.pyenv/versions/env_prefect2/lib/python3.8/site-packages/botocore/client.py", line 514, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/ubuntu/.pyenv/versions/env_prefect2/lib/python3.8/site-packages/botocore/client.py", line 901, in _make_api_call
request_dict = self._convert_to_request_dict(
File "/home/ubuntu/.pyenv/versions/env_prefect2/lib/python3.8/site-packages/botocore/client.py", line 962, in _convert_to_request_dict
request_dict = self._serializer.serialize_to_request(
File "/home/ubuntu/.pyenv/versions/env_prefect2/lib/python3.8/site-packages/botocore/validate.py", line 381, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Missing required parameter in input: "jobs"
from prefect_aws.client_waiter import client_waiter
from prefect_aws import AwsCredentials
aws_creds = AwsCredentials(region="us-east-2")
waiter = client_waiter(
client="batch",
aws_credentials=aws_creds,
waiter_name="JobCompleteOnSuccess",
waiter_definition={
"version": 2,
"waiters": {
"JobCompleteOnSuccess": {
"operation": "DescribeJobs",
"delay": 60,
"maxAttempts": 288,
# "description": "Wait until job status SUCCEEDED",
"acceptors": [
{
"argument": "Status",
"expected": "SUCCEEDED",
"matcher": "pathAny",
"state": "success"
}
]
}
}
}
)
Andrew Huang
01/05/2023, 5:14 AM