Hi, we migrated from prefect 1.0 to prefect 2.0. With this migration we are experiencing huge performance issues running the same flows as before. Interestingly, when keyboard interrupting a prefect flow on the CLI the task continues to run - with better performance. It seems this keyboard interrupt exits a prefect wrapper that is responsible for the performance issues. Attached is a screenshot of the error code when interrupting a prefect flow. Anybody experiences similar performance issues after the migration?
This issue happened when using the "Concurrent Taskrunner". Switching to the "Sequential Taskrunner" seems to work better
05/31/2022, 2:23 PM
Is your code simple enough to share or could you make an example around this for us to take a look at?
05/31/2022, 5:14 PM
We have created 1 flow which contained 1 parameter, 1 normal function and several (very similar) tasks with dependencies.
The tasks followed all the same scheme (a simple for loop with DB requests).
As described, this caused issues when using the sequential task runner. It works a lot better (by a factor of 4 faster) when running it with the sequential taskrunner.