Thang Le
02/26/2026, 6:49 PMMarvin
02/26/2026, 6:49 PMMarvin
02/26/2026, 6:55 PMentrypoint.
What does work well
- Store your flow code on S3 as a normal directory and deploy from it:
from prefect import flow
@flow
def my_flow():
return "hi"
if __name__ == "__main__":
my_flow.from_source(
source="<s3://my-bucket/path/to/project>", # requires s3fs installed in the worker env
entrypoint="flows/my_flow.py:my_flow",
).deploy(
name="s3-deployment",
work_pool_name="process-pool", # or your pool name
)
Notes:
- Ensure your worker environment has s3fs (or install prefect[aws]).
- This is the recommended way to keep code in S3 with Prefect 3.x.
If you must use a PEX
- There’s no native “PEX storage” or PEX entrypoint. You can hack around it with custom pull steps, but you’ll lose Prefect’s standard orchestration semantics (importing the flow object, parameter binding, retries/cancellation signals, rich state reporting, etc.).
- Example of a best-effort approach using pull steps in `prefect.yaml`:
pull:
- prefect.deployments.steps.pull_from_remote_storage:
url: <s3://my-bucket/artifacts/flow.pex>
- prefect.deployments.steps.utility.run_shell_script:
script: |
chmod +x flow.pex
./flow.pex # runs your binary; Prefect isn’t importing the flow here
Caveats:
- This runs your PEX as a separate process. The deployment’s flow run won’t be driven by Prefect’s engine unless your PEX itself connects to Prefect and manages state, which is non-trivial.
Recommended alternatives to PEX with Prefect
- Keep code on S3 as regular Python files and use from_source("s3://...") with a file-based entrypoint.
- Or package your code into a Docker image (you can even include your PEX inside) and use a Docker/Kubernetes work pool so the worker can import your flow module normally inside the container.
Would you share what’s driving the PEX requirement (e.g., packaging policy, hermetic builds)? Also, which worker/infrastructure are you using (Process, Docker, Kubernetes)? I can tailor a setup (S3-based or container-based) that fits your environment.