The error message you're encountering, "FutureCancelledError: flatten_json_parallel-0-4b5bf58794d24d18bc33bbf9cd64853c-1 cancelled for reason: scheduler-connection-lost," typically indicates a disconnection between your Prefect flow and the Dask scheduler. This can occur due to various reasons such as network issues, misconfigurations, or resource constraints.
Here are some potential solutions and considerations:
1.
Check Network Stability: Ensure that your network connection is stable. Any transient network issue could cause the client to lose connection with the Dask scheduler.
2.
Resource Availability: Verify if there are enough resources allocated for your Dask cluster. Insufficient resources could lead to unexpected cancellations.
3.
Dask Scheduler Configuration: Check your Dask scheduler configuration for any misconfigurations. Ensure that the scheduler is set up correctly to handle your workload.
4.
Retry Logic: Implement retry logic within your flow to handle transient issues. This may avoid flow failures due to temporary disruptions.
5.
Logs and Monitoring: Enable and monitor the logs for both Prefect and Dask. This can provide more insight into the failure and help identify specific issues with the connection.
For more detailed guidance on using Dask with Prefect 2.x, refer to
Prefect's Dask task runner documentation.
If the problem persists, you might want to visit Prefect's
GitHub issues page to see if others have encountered similar issues or report it if it appears to be a bug.