@Marvin Heya! So, we used to have individual "dev" Projects in Prefect 1, and a separate production Project. So like there's a version of Flow X on my dev Project, and another version of Flow X on the production Project (and ya can eventually merge changes from one into the other). How do we approximate this with the new Tagging system with just one Workspace? Can you have different Deployments of the same Flow that have different code for the flow? Do ya do it purely with tagging Tasks? Or will individual versions of flows need different names?

Ah, the joys of change. In Prefect 2.0, the concept of "Projects" has been replaced with "Workspaces" and "Tags". The "Tags" are used to organize your deployments and flow runs, and you can use these tags as filters in the Prefect UI. You can assign custom parameter values for flow runs based on the deployment, create ad-hoc flow runs from the API or Prefect UI, and upload flow files to a defined storage location for retrieval at run time. You can also specify run time infrastructure for flow runs, such as Docker or Kubernetes configuration. As for your question about having different deployments of the same flow with different code, you can package not only your flow code script but also any supporting files, including your custom modules, SQL scripts, and any configuration files needed in your project with remote storage blocks. And as for different versions of flows, until Prefect 2.0.4, tags were used to associate flow runs with work queues. Starting in Prefect 2.0.5, tag-based work queues are deprecated. Instead, work queue names are used to explicitly direct flow runs from deployments into queues. So, in essence, you can still have different versions of the same flow, but you'll need to use the new tagging system and workspaces instead of individual projects. It's a bit like trying to find your way around a new city after they've changed all the street names. But don't worry, you'll get the hang of it. Or not. I'm a robot, what do I care?

hmm yeah on this i think it always depends on how an org likes to use git do you have a monorepo situation?

@Nate Yep

whats the general structure like?

repo_root/
   project_1/
   ...

or like

repo_root/
   prd/
     project_1/
   stg/
     project_1/

or something else?

repo_root/
   project_1

and then the individual dev Projects live on different Git branches

gotcha, in that case I'd just want my github action to discover which branch it finds itself calling

prefect deploy

from, and attach a corresponding tag to that deployment (or even switch workspaces based on branch if youve got multiple i see now its 1 workspace)

@Nate Ah, okay! Just double-checking that "same Flow name, different Deployments with different code" is something that the system allows. my first attempt didn't quite work, so I wanted to make sure it was possible.

yep! prefect is entirely agnostic to your source code, and flow -> deployment being one -> many is totally a thing

Ah, sweet, thanks!

PS:

totally a thing

as long as the deployments have different names, so maybe

prefect deploy -n original-name-{branch}

or something

@Nate Hrm, having trouble getting it to work - I've got two Deployments with slightly different code (differentiated by one having a

_2

at the end). Tried running the first one, but it fails with this

PermissionError: [Errno 13] Permission denied: './../Hello World-ec2-ec2__cpu_2048__memory_14360_2'

is

name

being set in

base_args

uniquely for each deployment?

yep!

as in the deployment name, not the flow name

Same flow name, but one should be

ec2__cpu_2048__memory_14360

and the other should be

ec2__cpu_2048__memory_14360_2

if you look in the UI or something at each deployment, what is the

flowname/deployment

for each and what is the storage key for each?

v1

def make_deployment_args(
    flow_name: str,
    memory: int,
    user: t.Optional[str] = "dev",
    image: t.Optional[
        str
    ],
    cpu: t.Optional[int] = None,
    tags: t.List[str] = [],
) -> dict:
    full_ecs_args = make_ec2_args(memory, user, image, cpu)
    ecs_task_block = ECSTask(**full_ecs_args)
    resource_string = full_ecs_args["name"]

    return {
        "infrastructure": ecs_task_block,
        "name": resource_string,
        "work_queue_name": "default",
        "work_pool_name": "default-agent-pool",
        "path": f"{flow_name}-ec2-{resource_string}",
        "output": f"{flow_name}-ec2-{resource_string}.yaml",
        "apply": True,
        "tags": tags,
    }

version 2, same thing except

resource_string = full_ecs_args["name"]

->

resource_string = full_ecs_args["name"] + "_2"

sorry, itll be

entrypoint

in the UI

aha!

speaking as a prefect person, imo

path

is the most confusing thing about the infra block paradigm 🙂 personally i like to leave

path

alone whenever possible and just set

entrypoint

relative to the root of my storage block (but ik sometimes you have to mess with

path

)

ah, okay, I didn't realize it'd have a good default argument!

nope that shouldn't matter - that's a warning you shouldnt need to worry about (that we should clean up)

@Nate Hrm, path is

Hello World-ec2-ec2__cpu_2048__memory_14360

- any chance that the space is causing trouble?

hmm honestly not sure, are you still setting

path

yourself?

Nope, leaving it blank now

ah man, that's weird. sorry, i can pick this up with you tomorrow if you're still hitting issues - afk now

Thanks btw!