@Marvin prefect v2 flow submit 5000 tasks using dasktaskrunner. sometimes i see an old task runs again, despite being marked as Completed

It seems like you are encountering an issue with Prefect 2.x where tasks that have completed may be rerun unexpectedly when using the

DaskTaskRunner

. This is a known issue and can be related to several causes: 1. Worker Retirement: There are cases documented where Prefect does not handle rescheduling events or worker retirement well. This can result in tasks being marked as completed, but actually being rescheduled to run again if a worker retires. Check out this GitHub issue for more details: Prefect is unable to handle rescheduling events or worker retirement. 2. Race Conditions: Another issue might be related to race conditions where files required for tasks are not available at the time of execution. This is discussed in another GitHub issue: Path does not exist error. To address these issues, make sure: - You are using the latest version of Prefect, as newer releases may have fixed known bugs. - Consider adjusting your Dask cluster settings, such as ensuring workers do not retire during task execution and checking the task scheduling logic. - Explore adding more robust error handling and retries in your flow configurations to handle unexpected task restarts better. If you are still facing issues, consider raising the problem directly in the Prefect's community forums or checking if updating Prefect addresses your issue.