https://prefect.io
Join Slack
newbie / vague question: what is the recommended way for running multiple independent Prefect flows?...
# ask-community
d
newbie / vague question: what is the recommended way for running multiple independent Prefect flows? just looking for general ideas / best practices to incrementally go from a single flow defined in a Python script to arbitrarily many with no dependencies between them and separate schedules
k
Hey @Daniel Saxton, no recommended way…you just register and run them separate as long as your hardware can handle it? I think if you scale to multiple flows, just leave the agent on and the flow schedule will take care of execution.
Does that help you?
d
thanks @Kevin Kho, i think i may need to read up a bit more, but i believe i'm running the flow "agentless" (is that what happenes when you use 
prefect run
?) which may mean i should change that soon
would something like 
prefect run ... &
(sending to background) and 
prefect run ...
again be abusing the CLI a bit?
k
Yeah that is agentless execution. So for production you normally have an agent up and running that polls prefect cloud for flows to run and then will be responsible for deploying them.
I don’t know. I haven’t seen agentless execution in production yet, but you might be able to do it
d
ok, i think i'll work on making that change, thanks for the quick replies!
👍 1
@Kevin Kho thanks again for being so helpful, for now i actually ended up just kicking off multiple processes with 
prefect run
(i'm essentially using "Prefect-lite" at the moment with only the CLI on a single machine and no Prefect server, and then sending logs directly to another platform)...i'm wondering if this somehow has the possibility to interfere with Prefect's ability to parallelize flows (presumably the OS should still be sending separate processes to different cores if available, and then each Prefect process can "fan out" from there)? this may be more of a dask question if that's what Prefect is using under the hood
k
The default executor of Prefect is the LocalExecutor, which is single threaded and sequential, so if you deploy multiple of these, they should be fine if you have enough cores as you said. There is no Dask by default, but if you use the LocalDaskExecutor, it would try to occupy as many cores as possible so that is there you would run into flows competing for resources.
d
thanks!
k
Did you get agentless working for production btw? Am curious
d
what i've been working on is more or less is "production" in a sense (or the beginnings of it), and i would say it's working fine so far, but i'm not sure how sustainable it is yet
so far my goal has really just been to replace cron without adding "too much" more functionality yet, and i'd say Prefect has been very good at that
k
Ah ok gotcha
d
my suspicion is eventually we'll need to add the backend and perhaps the UI from Prefect cloud as well, i just lack the knowledge / time to do that yet 🙂
10 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by