https://prefect.io
Join Slack
Hello, I have high level design question: It is advisable to share the flow code between two deplo...
# ask-community
d
Hello, I have high level design question: It is advisable to share the flow code between two deployments of the same flow? details in thread
✅ 1
It looks like I could do this with the Python API:
Copy code
storage_block = S3.load("myfirstblock")

deploymentA = Deployment(
    flow_name="FooFlow",
    name="FooA",
    parameters={"arg", "A"}
    entrypoint="foo.py:foo"
    storage=storage_block,
    path="/foo_code"
)
deploymentA.apply()
deploymentA.upload_to_storage()

deployment = Deployment(
    flow_name="FooFlow",
    name="FooB",
    parameters={"arg", "B"}
    entrypoint="foo.py:foo"
    storage=storage_block,
    path="/foo_code"
)
deploymentB.apply()
k
Do you mean to ask if it is desirable to have two deployments for the same flow?
d
I am storing code in s3 for now. I am going to have, for example, 100-200 deployments per flow (each with different arguments) so I would like to avoid making a separate upload for each deployment. Is re-using code like this one of the intended ways Deployments are used, or am I doing off the rails?
it is desirable to have two deployments for the same flow?
I'm asking specifically about whether its acceptable to have two deployments share the exact same chunk of flow code stored in S3.
From the docs it seems like that having multiple deployments per Flow registration is by design.
k
Yes, deployment allows you to run a single flow with different parameters as explained here
d
Thanks, I read that. That is not what I'm asking.
k
However, you can create one deployment then override the default parameter with a custom parameter when running that deployment
d
I would like to have two deployments (for the same flow), each with separate sets of default args (that may be overridden), but only upload the code to s3 once.
k
You can skip uploading when building a deployment using the command 
--skip-upload
d
Great. Just wanted to make sure I wasn't violating an implicit design rule. Thanks.
2 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by