Hi!
I've had some problems with flows getting stuck after all tasks are finished and did not receive support so I've tried to recreate a minimal reproduceable example see code below.
What is happening for me:
The tasks can run fine, but after the tasks completes the memory usage explodes (goes to 100%). Depending on the size of the data the flow will either:
• get stuck after last finished task
• finish succesfully
• either of above intermittently
I can run the same flow perfectly fine in Prefect 1, it also runs a lot faster even when increasing data a lot. I attach a log of a unsuccesful prefect 2 run in the thread:
Hi Andreas. Sorry to hear about the issue. I consulted with our folks internally and we don’t have a quick fix. Are you able to file as an issue at https://github.com/PrefectHQ/prefect/issues?
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.