https://prefect.io logo
b

Ben Wolz

01/26/2021, 4:18 PM
Hi all, I'm currently running a simple flow with a Dask Executor that's spinning up a FargateCluster, but I'm getting a strange error that I can't find anywhere else online. When I run my flow with a DaskExecutor set up for a FargateCluster, I get a set of RunTimeErrors which say "IOLoop is closed". I've attached a .txt file of the code as well as the stack trace, if anyone has any advice it would be greatly appreciated! Thank you in advance
j

Jim Crist-Harif

01/26/2021, 4:51 PM
It looks like your flow ran successfully - the errors your seeing appear to be a bug in the shutdown process for dask-cloudprovider. I believe these are innocuous (prefect should shutdown your cluster explicitly, the errors are occurring because it's trying to shutdown again after the IOLoop has stopped). If you're a github user, would you mind opening an issue in the dask-cloudprovider repo for this? https://github.com/dask/dask-cloudprovider. You'll want to include your python and dask-cloudprovider versions. If not, I can open an issue, just let me know.
b

Ben Wolz

01/26/2021, 5:06 PM
@Jim Crist-Harif thanks for the explanation. That's what I assumed was going on, but it was hard to verify since I'm not too experienced with the intricacies of Dask. I'll open an issue, thank you for pointing the in the right direction!
Also, and I'll be sure to include this in the issue, it appears that the Dask cluster which is spun up does not get torn down. I've found a workaround to this problem by explicitly creating a Fargate Cluster and assigning a DaskExecutor to the address of said cluster's scheduler, which seems to work the same operationally.
j

Jim Crist-Harif

01/26/2021, 5:15 PM
Hmmm, that's a more severe issue. Prefect manually calls
cluster.close()
when running flows, so if the cluster isn't shutting down properly this would still be a bug in dask-cloudprovider. I haven't seen this issue, it may specific to your environment (or your cluster configuration).
b

Ben Wolz

01/26/2021, 5:16 PM
maybe it's an environment issue because the cluster config seems fairly similar to the defaults