Bob Colner
06/22/2022, 8:34 PMMichael Duncan
06/22/2022, 9:57 PMstate.result
from the flow which works locally but not in the cloud explained in this answer https://prefect-community.slack.com/archives/CL09KU1K7/p1653606914417649.
Is there a good way to collect exceptions from each task and report them in a state handler when the flow fails? We have have alerts on flow failures and we want to report the exceptions in the alertAbin Antony
06/22/2022, 10:09 PMAbin Antony
06/22/2022, 10:20 PMKen Nguyen
06/23/2022, 12:52 AMis_cancelled()
when looking here.Amol Shirke
06/23/2022, 2:06 AMyu zeng
06/23/2022, 3:44 AMluke b
06/23/2022, 5:57 AMDanilo Drobac
06/23/2022, 8:49 AMlocalhost:8000
.
I'm trying to connect to the instance (to make an API call) through a Python script, but I have no idea how to think about the authentication layer.
How do I send an API call to the server? The documentation suggests not to open the instance up to HTTP traffic but I'm guessing that's a requirement to be able to communicate with the API?Raviraja Ganta
06/23/2022, 9:58 AMsuccess_task
that will send a SQS message to my backend server indicating that the flow ran successfully. Now, I want to send a message when there is a failure in execution of task as well. Since the amount of tasks can be high (more than 10), is there a better to do error handling of the tasks?Dominik Wagner
06/23/2022, 10:21 AM2.0b7
and noticed that prefect orion kubernetes-manifest
is not a valid command. I assume I can just use the manifest from 2.0b6 and change the image version - so this is more of an FYI, since it’s still in the docs. Or is there something else I should be aware of here?Surya
06/23/2022, 10:58 AMJoshua Greenhalgh
06/23/2022, 11:01 AMZheng Cheng
06/23/2022, 11:51 AMZheng Cheng
06/23/2022, 12:34 PMjack
06/23/2022, 1:31 PMproject/
flow.py
scripts/
util.py
And in flow.py, there is this import statement:
from scripts.util import Util
Also in flow.py, using this storage:
with Flow('some-name', storage=S3(bucket='some-bucket')):
...
But when run from the cloud, it complains No module named "scripts.util"
redsquare
06/23/2022, 1:38 PMStéphanie Cérulli
06/23/2022, 2:00 PMStéphanie Cérulli
06/23/2022, 2:00 PMMichal Zawadzki
06/23/2022, 2:24 PMprefecthq/prefect:2.0b7-python3.10
to be exact); I was getting error code 255 at the apt-key add
step in my custom image. Adding `apt install gnupg`fixed the issue.Constantino Schillebeeckx
06/23/2022, 2:34 PMrectalogic
06/23/2022, 3:13 PMScott Aefsky
06/23/2022, 5:27 PMcapture_ID_batch, all_capture_metadata = get_unprocessed_capture_ID_batches(snowflake_tbl, experiment_id=10, batch_size = 500)
capture_loc_map = get_locations_for_captures(all_capture_metadata, s3_bucket)
#read the files from local disk given IDs, process and record proc jsons
process_batch.map(capture_ID_batch, unmapped(snowflake_tbl), unmapped(capture_loc_map), unmapped(intrazone_extractor), unmapped(dynamic_zone_finder))
The Prefect schematic shows that my first task `get_unprocessed_capture_ID_batches`is getting duplicated, and my flow is eventually failing because of a Dask scheduler timeout.
The last task starts running, but won't complete because the timeout kills it. I'm trying to understand a couple of things:
• Why is my first task showing up as if it's a mapped task?
• Why is the base node of that task not completing if each of the 2 instances of it are complete?
• Why is the scheduler timing out? If I run this flow with a small amount of data, it runs fine, but with a larger dataset I get this failure.
Thanks for any help you can provide!Nicholas Kan
06/23/2022, 5:29 PMJavier Ochoa
06/23/2022, 6:48 PMbotocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the GetParameter operation: User: arn:aws:sts::999999999999:assumed-role/MyRole-dev/12345459ba45458183ed3d1aa5112341 is not authorized to perform: ssm:GetParameter on resource: arn:aws:ssm:region:999999999999:parameter/XXX/value because no identity-based policy allows the ssm:GetParameter action
What is your suggestion here, or a possible solution for this?Jehan Abduljabbar
06/23/2022, 7:32 PMAmit Gupta
06/23/2022, 7:40 PMGuoying Qi
06/23/2022, 8:37 PMprefect.exceptions.PrefectHTTPStatusError: Client error '400 Bad Request' for url 'https://*******************************'
Response: {'detail': 'Request specified API version 0.5.0 but this server only supports version 0.6.0 and below.'}
For more information check: <https://httpstatuses.com/400>
Which version of the Prefect I should install on my local machine? thanks.Guoying Qi
06/23/2022, 8:54 PM{
"detail": "Not Found"
}
Guoying Qi
06/23/2022, 9:02 PM