Hi there, We are looking for azkaban alternative, evaluating airflow and prefect. At our daily batch computation flow, we have 800+jobs. I wonder that what is the max num of tasks that prefect can handle in single flow?

I’m not from the 😛refect: team but I don’t think there’s a limit, I guess it depends on the environment you’ll be running on. I myself run a flow that sometimes has 5k+ tasks in it. Here’s a screenshot from the latest run (1.6k tasks)

Hi @kursat.t, there's no enforced limit on the number of tasks. Users regularly create and run flows with 100s to 1000s of tasks (usually mapped tasks).

Hi @Avi A, Thank you for your quick answer. Is 1610 the iteration number or actual explicit task count? How is web ui handles 1000+tasks. Also my all tasks are ~long running ELT tasks, generates and sends queries to an external query execution engine.

Hey, it’s task count. and I remember these visualizations from the time I used Luigi (not for the same workflow). The prefect UI doesn’t render all the mapped tasks like Azkaban does, but TBH it does freeze the browser sometimes and I have to refresh it, not sure why it happens (i’m hosting it myself, not using prefect cloud at the moment)