https://prefect.io logo
m

Matt Alhonte

08/22/2023, 11:40 PM
Heya! So, we used to have individual "dev" Projects in Prefect 1, and a separate production Project. So like there's a version of Flow X on my dev Project, and another version of Flow X on the production Project (and ya can eventually merge changes from one into the other). How do we approximate this with the new Tagging system with just one Workspace? Can you have different Deployments of the same Flow that have different code for the flow? Do ya do it purely with tagging Tasks? Or will individual versions of flows need different names?
j

Jamie Zieziula

08/23/2023, 1:10 AM
Hi @Matt Alhonte! classic answer for you is it depends πŸ™‚ you can definitely have multiple deployments (one for dev and one for prod) that share the same underlying flow code and pass different parameters to each, have them run in separate work pools (cloud environments), etc. If you want to test the flow code before merging to main, you could setup your deployment to pull from a β€œdev” like branch, or deploy the deployment locally
also of course you could separate by workspace which would allow you the most separation between environments
m

Matt Alhonte

08/23/2023, 1:11 AM
@Jamie Zieziula Can the underlying code be different though? ie, like literally different code in the body of the Flow or its Tasks?
Would rather not spin up different Workspaces due to increased costs & complexity (multiple keys that need to be piped into various other systems, for instance)
j

Jamie Zieziula

08/23/2023, 1:14 AM
Do you mean different code like totally different code? or changes you want to test in a dev environment?
m

Matt Alhonte

08/23/2023, 1:14 AM
Including totally different code, yeah
Same Flow name, same name of Flow function, different code.
Makes it a lot easier to merge changes
j

Jamie Zieziula

08/23/2023, 1:15 AM
are both versions of those code meant to live on main?
m

Matt Alhonte

08/23/2023, 1:15 AM
No, they'd live on different git branches
j

Jamie Zieziula

08/23/2023, 1:15 AM
oh i see
and ya can eventually merge changes from one into the other
yep so you can do as i mentioned above - youll have two deployments, with the associated env-specific configuration. youll need the pull step of your dev deployment to point to the dev-branch & the pull step of your pd deployment to point to the prd-branch
m

Matt Alhonte

08/23/2023, 1:16 AM
The way we did it in Prefect 1 was you'd make a change on your branch -> CI/CD runs & rebuilds your Image & re-registers your versions of the flows onto your Project
j

Jamie Zieziula

08/23/2023, 1:18 AM
you could accomplish the same now, it might just take a bit more cicd magic to get it to work for your dev branches (unless you have a long lived branch)
the merge to main workflow should be simple though
m

Matt Alhonte

08/23/2023, 1:20 AM
@Jamie Zieziula Does that work well with deploying the
script
way? Right now the Prefect 2 workflow has something like this at the bottom of every Flow file
Copy code
for memory in [15360, 30720, 63398, 126899]:
        base_args = make_deployment_args(flow_name1, memory, user="matt")
        storage1 = S3Bucket.load(<name>)
        deployment1 = Deployment.build_from_flow(
            flow=nb_flow, storage=storage1, **base_args
        )
j

Jamie Zieziula

08/23/2023, 1:20 AM
okay sorry sorry, i should have asked! i assumed you were using the new version of deployments & workers
m

Matt Alhonte

08/23/2023, 1:21 AM
For the test I'm running, I've got one Deployment called
ec2__cpu_2048__memory_14360
and one called
ec2__cpu_2048__memory_14360_2
j

Jamie Zieziula

08/23/2023, 1:21 AM
i am not as familiar with that deployment methodology, ill ask a peer of mine if they can way in
πŸ‘ 1
m

Matt Alhonte

08/23/2023, 1:21 AM
When I try to run the first one, it fails with
PermissionError: [Errno 13] Permission denied: './../Hello World-ec2-ec2__cpu_2048__memory_14360_2'