https://prefect.io
Join Slack
when I do a `prefect deploy --all` and I have multiple flows defined in my `prefect.yaml` will the b...
# ask-community
c
when I do a 
prefect deploy --all
and I have multiple flows defined in my 
prefect.yaml
will the build/push steps get executed only once for all flows (since they all share those actions), or does build/push get executed for every flow? From my initial testing, I'm seeing:
Copy code
> Running push_to_s3 step...
executed once for every flow (I haven't tested 
build
yet), which doesn't quite make sense to me since a single push should be enough for all flows.
āœ… 1
k
it does run once for every deployment, but you can override those actions at the deployment level. So if you only wanted to push once, you could ā€¢ set 
push: null
on all but one of the deployments in your file, and let the one remaining deployment use the global 
push
, or ā€¢ set the global 
push
to 
null
and have 
push_to_s3
on just one deployment
šŸ˜¢ 1
c
does the same apply to 
build
?
k
yep
c
thanks for the info, gives me what I need to implement a workaround
k
under what circumstances is 
prefect deploy --all
being run? maybe we can come up with something that feels a little less silly than my generic answer
c
every merge to 
master
should deploy all flows; all the flows use the same S3 storage and the same docker image.
i guess i'm having a hard time wrapping my brain around why you wouldn't want to deploy all your flows, or why you'd want your flows to have different storage
i'm trying to abstract away deployment details from other teams that will be developing flows
k
I think about it like this: deploying flows is a range of behaviors, and what's necessary to do when you merge depends on what in your repo has changed
minimally, it's updating deployment metadata like parameters or job variables. in this case, only the prefect.yaml changed and that's not something your deployment actually needs to run. so an upload to s3 isn't a requirement
maximally, your code changed, your dependencies changed, and your deployment metadata changed, so you need to build a new image, push your code to remote storage, and update your deployments
depending on your approach to project/repo management and cicd, it might not always make sense to rely on prefect.yaml's build and push steps
c
or we just build and push once for all flows šŸ™‚ and call it a day
ok, that all makes sense to me - thanks for all the info.
hey @Kevin Grismore wanted to follow up with this; my prefect.yaml file looks like:
Copy code
name: foo
prefect-version: 2.13.7

build: null


push:
  - prefect_aws.deployments.steps.push_to_s3:
      id: push_code
      requires: prefect-aws>=0.3.4
      bucket: '{{ $STORAGE_BUCKET }}'
      folder: '{{ $STORAGE_PREFIX }}'
pull:
  - prefect_aws.deployments.steps.pull_from_s3:
      id: pull_code
      requires: prefect-aws>=0.3.4
      bucket: '{{ push_code.bucket }}'
      folder: '{{ push_code.folder }}'


deployments:
  - name: DE
    push: null
    version: null
    tags: [DE]
    description: null
    schedule: {}
    entrypoint: hs_de_workflows/flows/hello_world.py:hello_world
    parameters: {}
    work_pool:
      name: ecs-staging
      work_queue_name: null
      job_variables:
        memory: 1024
        container_name: hello_world
        cpu: 512
  - name: DE
    push: null
    version: null
    tags: [DE]
    description: null
    schedule: {}
    entrypoint: hs_de_workflows/flows/hello.py:hello
    parameters: {}
    work_pool:
      name: ecs-staging
      work_queue_name: null
      job_variables:
        memory: 1024
        container_name: hello_world
        cpu: 512
you'll notice i've set 
push: null
on both flows; yet when I deploy, I still two logging statements like
Running push_to_s3 step
which tells me it's still pushing; what am I missing?
note that your second suggestion:
ā€¢ set the global 
push
to 
null
and have 
push_to_s3
on just one deployment
does work (only a single push occurs)
k
it looks like I may have given some bad advice, sorry. setting the per-deployment step override to null is probably the same as not having it at all, which would cause the global setting to be used
which makes perfect sense in retrospect
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by