Thread for your thoughts: destroy Environments 🔥 What do you like/use from Prefect Environments? We are thinking of getting rid of them in favor of bloating the agents more with more configuration. The reason for this is to attempt to make it easier to make a choice on where to deploy. For example, today if you use a Kubernetes Agent you probably are using one of a limited list of environments (such as KubernetesJobEnvironment or DaskKubernetesEnvironment); if you are using the Fargate Agent you are probably only using the Fargate Task Environment; etc. So basically we think they are not as uncoupled as we originally thought and it might be easier for people to use them if we just had single configuration on an Agent for each of these combinations. What do you think? Is there something from the Environment config that you particularly like or use now that you would be worried would be lost if we removed them?

@Laura Lorenz (she/her) Throwing it out there: How about the other way? One agent, many environments. This would allow for "easy" switching/experimenting between environments as the workflow develops. From a user perspective this would simplify managing agents i.e logging, managing prefect versions and deploying the agents.

I think I agree with @itay livni here. Agents, as they currently are, can deploy multiple flows. Even if the environment type is the same, environment configs can differ. Without an environment-like object, how will things like

dask-address

or

nthreads

be specified? I understand that a singular agent feels bloated with features and responsibilities. But a 1-to-1 mapping between a registered flow and its infrastructural requirements makes so much sense, and the amount of customization to these requirements calls for a construct that contains these requirement specifications.

If environments were done away with (or moved to the agent), how would core only people be able to utilize them? I think requiring an agent for core only folks is a decision that shouldn’t be taken too lightly, but could be a beneficial one. It could familiarize people with the concept of agents prior to touching server / cloud, but increases the number of things that need to happen as an on ramp. One thing I’ve always liked about Prefect is the incredibly minimal on ramp to use some core functionality, and I’d be hesitant about changing that. I’m personally of the mindset of enjoying all flow related configuration options being “on the flow” itself when I’d be defining and registering a flow. I think a current pain point people get stuck on when transitioning strictly from core to core + server or core + cloud is the agent already, and adding more onto that pain point could potentially make it worse (could also make it better by forcing people to be familiar with the process earlier, so who knows).

Ok interesting thoughts thank you @itay livni @emre and @Alex Cano!!! To your point on

nthreads

etc @emre, we have been moving that type of configuration to the DaskExecutors, which is part of why environments started to feel a bit light to us (land thus potentially superfluous). Definitely hear you all on the 1:1 flow:environment niceness when running on heterogeneous infra. Definitely don’t want to take away the possibility of that, though we do think that is a bit of a rare case (feel free to debunk me on this!) Tbh @Alex Cano I don’t think we really consider environments usable core-only 🤔 digging around the source code right now I don’t think they do anything in flow.run(). I will double check with our sages but if you are using environments in core-only I am very interested to hear the deets! Definitely feel you that throwing the agent in there alone is a lot of the upgrade pain. All in all I am taking these thoughts to our internal drawing board for sure!

Super real possibility environments are cloud only! I am currently not using them with strictly core, but made an assumption because I didn’t want to get check the code 😬. If it is a cloud only feature (in the sense that core just doesn’t use them), I see no harm in moving them around!

Ok gotcha, ya I think it’s a bit confusing because environments are an initialization argument to a Flow since you might need it serialized on it for orchestration layer purposes, but I don’t think the flow can do anything with it itself like in flow.run(). will triple check ofc 🙂