<@ULVA73B9P>, can you deploy pex files to s3 and d...
# marvin-ai
t
@Marvin, can you deploy pex files to s3 and deploy that to prefect?
m
thought for 362.6 seconds
Short answer: not directly. Prefect 3.x does not have first-class support for deploying a PEX artifact from S3 and importing it as a flow. Workers expect importable Python code (a file or module path) they can load via the deployment’s
entrypoint
. What does work well - Store your flow code on S3 as a normal directory and deploy from it:
Copy code
from prefect import flow

  @flow
  def my_flow():
      return "hi"

  if __name__ == "__main__":
      my_flow.from_source(
          source="<s3://my-bucket/path/to/project>",   # requires s3fs installed in the worker env
          entrypoint="flows/my_flow.py:my_flow",
      ).deploy(
          name="s3-deployment",
          work_pool_name="process-pool",             # or your pool name
      )
Notes: - Ensure your worker environment has
s3fs
(or install
prefect[aws]
). - This is the recommended way to keep code in S3 with Prefect 3.x. If you must use a PEX - There’s no native “PEX storage” or PEX entrypoint. You can hack around it with custom pull steps, but you’ll lose Prefect’s standard orchestration semantics (importing the flow object, parameter binding, retries/cancellation signals, rich state reporting, etc.). - Example of a best-effort approach using pull steps in `prefect.yaml`:
Copy code
pull:
    - prefect.deployments.steps.pull_from_remote_storage:
        url: <s3://my-bucket/artifacts/flow.pex>
    - prefect.deployments.steps.utility.run_shell_script:
        script: |
          chmod +x flow.pex
          ./flow.pex  # runs your binary; Prefect isn’t importing the flow here
Caveats: - This runs your PEX as a separate process. The deployment’s flow run won’t be driven by Prefect’s engine unless your PEX itself connects to Prefect and manages state, which is non-trivial. Recommended alternatives to PEX with Prefect - Keep code on S3 as regular Python files and use
from_source("s3://...")
with a file-based
entrypoint
. - Or package your code into a Docker image (you can even include your PEX inside) and use a Docker/Kubernetes work pool so the worker can import your flow module normally inside the container. Would you share what’s driving the PEX requirement (e.g., packaging policy, hermetic builds)? Also, which worker/infrastructure are you using (Process, Docker, Kubernetes)? I can tailor a setup (S3-based or container-based) that fits your environment.