https://prefect.io logo
Title
m

Mikael

08/14/2020, 7:13 AM
If I am to load 50 tables from 1 db to another. Would you implement that as 1 flow per table or 1 flow mapped over all tables? I guess if I make a loop to register 1 flow per table it would be easier to rerun only one table. And maybe easier with debugging. But it would be more cluttered in the UI.
j

Julian

08/14/2020, 7:40 AM
Keep in mind, that when you are mapping over all tables with e.g. Dask Executor, Prefect will try to parallalize this taks and potentially open 50 db-Connections. Aside from that I would personally prefer a single flow, but I would also like to hear the suggestion of the prefect maintainer 🙂
m

Mikael

08/14/2020, 8:25 AM
Thanks for replaying. Is there a way to limit the number of parallel tasks. On flow level or in the Dask executor maybe.
j

Jacob Blanco

08/14/2020, 9:20 AM
There is a Task Concurrency limit in Prefect Cloud if you have it enabled on your account.