Max Jackson
04/04/2023, 6:54 PMwait_for_flow_run
from Prefect 1? I'd like to have a flow of flows which stops in a Failed state if one of its subflows fails, but I'm not sure how best to implement this in Prefect 2.flapili
04/05/2023, 1:06 PMflapili
04/06/2023, 8:57 AMMax Jackson
04/06/2023, 5:35 PMmy_flow_run = prefect.deployments.run_deployment(name='my_deployment')
, then throwing an error if my_flow_run.state_name=='failed'
. I'd like to coordinate them asyncronously, with something like
my_flow_run_1, my_flow_run_2 = await asyncio.gather(
run_deployment(
name=f"my_deployment_1"),
run_deployment(
name=f"my_deployment_2")
)
throw_error_if_flow_run_failed(my_flow_run_1)
throw_error_if_flow_run_failed(my_flow_run_2)
then I'd like to continue the rest of the main flow sequentially.
Am I going about this the right way? is there a built-in solution or will I need to hand-roll something?Jafar A
04/07/2023, 3:59 PMJon
04/10/2023, 8:51 PMflow_run_name
in prefect 1. looks like the flow run name gets updated in the client in RenameFlowRun.run
, but the prefect context doesn't. is there an easy way to update the prefect context's flow_run_name
?Ying Ting Loo
04/14/2023, 4:05 AMJenia Varavva
04/17/2023, 10:46 PMgs://<bucket>/<commit hash>
2. For each of the cartesian product of Flow ⅹ Env (e.g. A-prod
, B-qa
) have:
a. A separate storage block pointing to a different subdir of gs://<bucket>/
b. A docker image label pointing to the image of the same commit.
3. The release to an environment would then include: updating the storage block to point to a different dir + updating the container image tag. Each deployment would also have an infra_override
pointing it at its specific image tag.
A couple of concerns I’m having are: storing all code/image versions (might be a good thing?). Maintaining potentially a large number of storage blocks (one for each of flow x env). Any thoughts?Jafar A
04/21/2023, 5:44 PMenabled
so this error shouldn't happen as per AWS I don't need to set up NAT gateway to allow my subnets to talk to ECR through the internet, or have to setup ECR VPC Endpoints.
Also, security group inbound and outbound is set for all ports and there is no secret manager used.
Submission failed. prefect_aws.ecs.TaskFailedToStart: ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve ecr registry auth: service call has been retried 3 time(s): RequestError: send request failed caused by: Post "<https://api.ecr.us-west-2.amazonaws.com/>": dial tcp 34.223.27.182:443: i/o timeout. Please check your task network configuration. See Less
Alvaro Durán Tovar
04/21/2023, 7:44 PMflows/api_flow.py
and the entrypoint is ./flows/api_flow.py:call_api
.
The error comes when triggering the flow to run it on kubernetes.
Failes with this error:
prefect.exceptions.ScriptError: Script at 'flows/api_flow.py' encountered an exception: FileNotFoundError(2, 'No such file or directory')
Andy Dienes
04/27/2023, 1:40 PMutils
I want to be common to all projects, but i want to develop and deploy projA
and projB
separately. I'm using the s3 recipeYing Ting Loo
05/04/2023, 11:12 AMFlow could not be retrieved from deployment. FileNotFoundError: [Errno 2] No such file or directory
I am suspecting it is not looking in the right agent.
My questions:
1. How does prefect server knows which agent to search from when a run happens?
2. How to troubleshoot to check if a flow is trying to get the code from the right agent?Nelson Griffiths
05/06/2023, 12:23 PMCloud Run Job
block, save it, and reuse if in building our python Deployment
objects.
Now, it seems that a lot of the infrastructure is taken over by the work pool. If I create a cloud run work pool
, what is the purpose of a cloud run block
? I already have all of my infrastructure existing in code now and I don't really see why I should want to use a cloud run block
. Other things like credentials still make a lot of sense for blocks. But some of them seem to be replaced by the new projects + work pools workflow.
Am I missing something here? Or does anyone have some good insights into this they would be willing to share?Andy Dienes
05/08/2023, 3:05 PMJafar A
05/16/2023, 7:52 PM_retrieve_server_version raise DockerException( docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory')) The above exception was the direct cause of the following exception: RuntimeError: Could not connect to Docker.
Abhinav Chordia
06/02/2023, 4:34 PMpython_repo /
projectA /
src/projectA/flows/...
pyproject.toml
projectB /
src/projectB/...
pyproject.toml
projectA depends on projectB and I’d like to deploy all live changes within the python_repo.
What is the recommended approach for doing this?Adrien Besnard
06/05/2023, 10:28 AMDeployment
from Python code, but I need to set a custom PREFECT_API_URL
but not through an environment variable. Is there a way to update the Settings
directly from code?Bebeto Nyamwamu
06/19/2023, 10:21 AMWT
06/20/2023, 7:52 PMBebeto Nyamwamu
06/21/2023, 7:00 AMArjun Krishnamurthy
06/26/2023, 12:28 PMYing Ting Loo
07/18/2023, 6:30 AMNelson Griffiths
07/18/2023, 8:57 PMEdmondo Porcu
08/14/2023, 8:51 PMJon
08/17/2023, 5:12 PMDeivid
08/24/2023, 12:21 AMcrashed
state for a minute or so and then I changes to running
.
Is that the expected behaviour?Marty Ko
09/06/2023, 10:15 AMNarcissa Morton
09/20/2023, 9:17 AMYing Ting Loo
09/22/2023, 7:50 AMdeployment.build()
and deployment.apply
would be registered in the prefect server even if the cicd container is not going to be running the flow(not a long time running prefect agent)?Andreas Nord
09/25/2023, 2:09 PM