I've been working through the tutorial and ran into a Dask KilledWorker error and was hoping someone might be able to point me to a solution or give me some pointers on how to debug this.I can start dask-scheduler and 2 dask-workers to successfully run the "Deployment: Dask" example. https://docs.prefect.io/core/tutorials/dask-cluster.htmlI can run the "Advanced Features" example through the DaskExecutor section where the DaskExecutor starts up the dask-cluster and process the URLs in 20% of the single threaded mode. https://docs.prefect.io/core/tutorials/advanced-mapping.html#outlineIf I add the scheduler address to the executor parameters
I get the KilledWorker error.
ERROR - prefect.Flow: x-files | Unexpected error occured in FlowRunner: KilledWorker('retrieve_url-a032a3d8-fa89-4131-ad4c-4f3c5fbd282c', <Worker 'tcp://192.168.86.23:58923', name: tcp://192.168.86.23:58923, memory: 0, processing: 1>)By commenting things out I'm down to the offending line being
html = requests.get(url)
in the retrieve_url function.Any thoughts?
2 years ago
Hi @Nate Atkins, KilledWorker is often a result of a worker running out of memory or hitting a segfault. (Dask is preventing the task from running again because it has previously killed a worker.) This link has more details: https://stackoverflow.com/questions/46691675/what-do-killedworker-exceptions-mean-in-dask Can you watch the Dask dashboard Workers tab for worker memory while your Flow runs? (Usually available on port 8787, e.g. http://localhost:8787 if you're running the Dask scheduler on your laptop.) Also see if there's any relevant output from the Dask workers.