https://prefect.io logo
Title
k

kursat.t

09/15/2020, 2:26 PM
Hi there, We are looking for azkaban alternative, evaluating airflow and prefect. At our daily batch computation flow, we have 800+jobs. I wonder that what is the max num of tasks that prefect can handle in single flow?
a

Avi A

09/15/2020, 2:38 PM
I’m not from the 😛refect: team but I don’t think there’s a limit, I guess it depends on the environment you’ll be running on. I myself run a flow that sometimes has 5k+ tasks in it. Here’s a screenshot from the latest run (1.6k tasks)
j

Jim Crist-Harif

09/15/2020, 2:46 PM
Hi @kursat.t, there's no enforced limit on the number of tasks. Users regularly create and run flows with 100s to 1000s of tasks (usually mapped tasks).
👍 1
k

kursat.t

09/15/2020, 3:07 PM
Hi @Avi A, Thank you for your quick answer. Is 1610 the iteration number or actual explicit task count? How is web ui handles 1000+tasks. Also my all tasks are ~long running ELT tasks, generates and sends queries to an external query execution engine.
an example from azkaban workflow scheduler 🙂
a

Avi A

09/15/2020, 3:21 PM
Hey, it’s task count. and I remember these visualizations from the time I used Luigi (not for the same workflow). The prefect UI doesn’t render all the mapped tasks like Azkaban does, but TBH it does freeze the browser sometimes and I have to refresh it, not sure why it happens (i’m hosting it myself, not using prefect cloud at the moment)