Scott Pedersen
08/05/2024, 10:43 PMDeployment.build_from_flow
handle pushing and pulling flow code to remote storage like an S3Bucket storage block whereas Flow.from_source().deploy()
will only pull flow code from remote storage?Marvin
08/05/2024, 10:43 PMDeployment.build_from_flow
, the current working directory is uploaded to the remote storage, which includes all relevant files unless specified otherwise.
* Flow.from_source().deploy():
- This method appears to pull flow code from remote storage but does not inherently push or upload the flow code to remote storage during deployment.
- It seems optimized for running flows that have been defined and are stored in a remote repository, where it pulls the necessary code to run the flow.
For more information, check out:
- Official documentation on Prefect Deployments: Prefect Deploy Documentation
- GitHub issue detailing remote storage behavior for `Deployment.build_from_flow`: GitHub Issue #13140
- GitHub discussions on `Flow.from_source().deploy()`: GitHub Issue #11546
If you need to ensure both uploading and pulling of flow code in a seamless manner, Deployment.build_from_flow seems to be the more robust choice.Eric Ramirez
09/17/2024, 3:25 AMEric Ramirez
09/17/2024, 3:38 AMScott Pedersen
09/17/2024, 3:38 AMFlow.from_source().deploy()
. One thing to note is that Deployment.build_from_flow
would upload flow code to S3 (in our case) and download the flow code as well. Flow.from_source().deploy()
will download your flow code from storage but not upload. Originally we were using the S3 block and tried the S3 bucket block, however, we had issues so we ended up writing our own logic to upload our flow code to S3.Scott Pedersen
09/17/2024, 3:39 AMEric Ramirez
09/17/2024, 3:39 AMScott Pedersen
09/17/2024, 3:40 AMEric Ramirez
09/17/2024, 3:40 AMScott Pedersen
09/17/2024, 3:42 AMflow_instance = Flow.from_source(
source=s3_bucket_name,
entrypoint="path/to/flow/my_flow_code.py:flow")
Eric Ramirez
09/17/2024, 3:43 AMScott Pedersen
09/17/2024, 3:43 AMEric Ramirez
09/17/2024, 3:46 AMEric Ramirez
09/17/2024, 3:46 AMScott Pedersen
09/17/2024, 3:48 AM