https://prefect.io logo
Title
p

Patrick Tan

11/10/2022, 6:51 PM
I am testing deploying flow to S3 using storage block in prefect 2.0. When I start running flow using Prefect Cloud UI. The agent acknowledged flow run submitted, but wait for 3+ minutes before it starts executing flow run. Does the agent download code in S3 before it starts the execute flow run?
1
z

Zanie

11/10/2022, 6:56 PM
Hi Patrick! Yeah the flow will only move to a running state when your code is about to be executed.
Before that, we download your code and check in with the API to ensure that the flow is ready to run.
p

Patrick Tan

11/10/2022, 7:03 PM
Typically I starts 30-40 flow run concurrently. Isn't it redundant to download same code for every flow run?
z

Zanie

11/10/2022, 7:04 PM
Perhaps there’s some room for optimization there! Generally people are running their flows on isolated infrastructure like containers.
p

Patrick Tan

11/10/2022, 7:29 PM
Currently we run our flow in one AWS ECS container, The purpose of moving storage to S3 is to scale with running multiple containers. However, Development is in macbook. I tested running agent in an empty folder and flow run failed, I expected code to be download to the empty folder but I do not see anything there. Where does the code download to?
z

Zanie

11/10/2022, 7:30 PM
It gets downloaded to a temporary directory
To provide isolation for each flow run
p

Patrick Tan

11/10/2022, 7:33 PM
How do you handle PYTHONPATH?If it is in a temporary directory, I cannot specify PYTHONPATH
z

Zanie

11/10/2022, 7:42 PM
I believe we push the temporary directory onto the PYTHONPATH
You can add paths relative to the flow’s execution directory using
./
e.g.
./my-path
.