I noticed somewhat inconsistent performance results. If I run the code in docker without prefect it takes around 154 seconds but when I run it in prefect it takes almost 900 seconds. However, if I explicitly set the task runner to sequential, then it takes ~ 180 seconds. Is my flow doing something wrong? Why does the concurrent task runner take so long? Thanks!
z
Zanie
06/14/2022, 2:45 PM
We’ll need more details about you flow/tasks, I’m surprised to see this behavior. If you can find out where the time is being spent, that’d be really helpful.
Zanie
06/14/2022, 2:45 PM
(or provide a minimal example that I can profile)
Zanie
06/14/2022, 2:47 PM
Is it still slow without those
wait
calls?
p
Paco Ibañez
06/14/2022, 2:49 PM
the first task is always super quick, the second one (which uses a language model) is the one that is really slow
Paco Ibañez
06/14/2022, 2:50 PM
but if I set the task runner to sequential it performs way better
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.