Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

newbie / vague question: what is the recommended way for running multiple independent Prefect flows? just looking for general ideas / best practices to incrementally go from a single flow defined in a Python script to arbitrarily many with no dependencies between them and separate schedules

Hey <@U010VT1UWNA>, no recommended way…you just register and run them separate as long as your hardware can handle it? I think if you scale to multiple flows, just leave the agent on and the flow schedule will take care of execution.

thanks <@U01QEJ9PP53>, i think i may need to read up a bit more, but i believe i'm running the flow "agentless" (is that what happenes when you use `prefect run`?) which may mean i should change that soon

would something like `prefect run ... &amp;` (sending to background) and `prefect run ...` again be abusing the CLI a bit?

Yeah that is agentless execution. So for production you normally have an agent up and running that polls prefect cloud for flows to run and then will be responsible for deploying them.

I don’t know. I haven’t seen agentless execution in production yet, but you might be able to do it

ok, i think i'll work on making that change, thanks for the quick replies!

<@U01QEJ9PP53> thanks again for being so helpful, for now i actually ended up just kicking off multiple processes with `prefect run` (i'm essentially using "Prefect-lite" at the moment with only the CLI on a single machine and no Prefect server, and then sending logs directly to another platform)...i'm wondering if this somehow has the possibility to interfere with Prefect's ability to parallelize flows (presumably the OS should still be sending separate processes to different cores if available, and then each Prefect process can "fan out" from there)? this may be more of a dask question if that's what Prefect is using under the hood

The default executor of Prefect is the LocalExecutor, which is single threaded and sequential, so if you deploy multiple of these, they should be fine if you have enough cores as you said. There is no Dask by default, but if you use the LocalDaskExecutor, it would try to occupy as many cores as possible so that is there you would run into flows competing for resources.

Did you get agentless working for production btw? Am curious

what i've been working on is more or less is "production" in a sense (or the beginnings of it), and i would say it's working fine so far, but i'm not sure how sustainable it is yet

so far my goal has really just been to replace cron without adding "too much" more functionality yet, and i'd say Prefect has been very good at that

my suspicion is eventually we'll need to add the backend and perhaps the UI from Prefect cloud as well, i just lack the knowledge / time to do that yet :slightly_smiling_face: