Hi prefecters I have a question about prefect core no Prefec

Question

Hi prefecters I have a question about prefect core no Prefect Server no Prefect Cloud I m trying to understand Flow concurrency Is the best way to handle a stream of input data to run multiple flows in their own thread as the data comes in

Steve R · Answer

alternatively I created a list of dask distributed events that can be waited on but I m not crazy about this solution

Steve R · Answer

So running multiple flows backed by the same executor is an alternative option

Kevin Kho · Answer

I think this is pretty hard without a backend because the standard way to spin up a flow is using the `create flow run` task which triggers a new flow run from the backend I don t even know if you can achieve concurrent flows using core I think the thread gets occupied with one flow and might not kick off other ones The Prefect agent than connects to the backend handles this for you I think this is a scenario where you may have to use Dask to orchestrate Prefect like you are suggesting but then I m wondering what Prefect gives you here

Steve R · Answer

Thanks I ll respond in a bit

Steve R · Answer

gt Concurrent flow runs are not supported by `flow run ` gt `flow run ` is a convenient way to run a flow on schedule but it does not support concurrent flow runs It will wait for a run to completely finish including things like tasks that require retries before starting the next run However Prefect schedules never return start times in the past This means that if a flow run is still running when another flow run is supposed to start the second flow run won t happen at all If you require concurrent runs in a local process consider using the lower level `FlowRunner` classes directly from <https docs prefect io core concepts flows html running a flow on schedule>

Kevin Kho · Answer

So Core alone is not intended for production but if you use it with Cloud or Server then you can run concurrently because it s not the same `flow run `

Steve R · Answer

Thanks Kevin Prefect Core is attractive as an richer object model abstraction over Dask on Windows It has paid support options and is cloud ready as that need may arise

Steve R · Answer

I was mostly interested in this line

Steve R · Answer

gt If you require concurrent runs in a local process consider using the lower level `FlowRunner` classes directly

Kevin Kho · Answer

I have never seen anyone do that and I don t think i d recommended working with that as working with the FlowRunner is pretty hard to read

Steve R · Answer

so it seems like structuring our problem in a way that operations are performed within a single flow run if using only prefect core is the way to go

Steve R · Answer

so like input could be a list of URLs to process even if some of the URLs will not return until the data is available

Steve R · Answer

because a prefect flow likes a list of work as input

Kevin Kho · Answer

I think you tagged a different Kevin but no worries That will work and mapping will help you If the data is not ready you can poll inside the task or just set a high number of max retries

Steve R · Answer

thanks I see this in the pipe as well

Kevin Kho · Answer

No no Not in the pipe lol Those are old But either way these would require a backend beyond Prefect Core

Kevin Kho · Answer

<https medium com the prefect blog event driven workflows with aws lambda 2ef9d8cc8f1a|This> is our paradigm for event based flows

Kevin Kho · Answer

what is your picture looks like masks over nuclei

Steve R · Answer

whole cell segmentation l think labelling live dead cells

Steve R · Answer

we do the things like that <https www moleculardevices com >

Kevin Kho · Answer

ahh I see