Nate Atkins

02/07/2020, 7:43 PM
I've been working through the tutorial and ran into a Dask KilledWorker error and was hoping someone might be able to point me to a solution or give me some pointers on how to debug this. I can start dask-scheduler and 2 dask-workers to successfully run the "Deployment: Dask" example. I can run the "Advanced Features" example through the DaskExecutor section where the DaskExecutor starts up the dask-cluster and process the URLs in 20% of the single threaded mode. If I add the scheduler address to the executor parameters
executor = DaskExecutor(address="<tcp://>")
I get the KilledWorker error. ERROR - prefect.Flow: x-files | Unexpected error occured in FlowRunner: KilledWorker('retrieve_url-a032a3d8-fa89-4131-ad4c-4f3c5fbd282c', <Worker 'tcp://', name: tcp://, memory: 0, processing: 1>) By commenting things out I'm down to the offending line being
html = requests.get(url)
in the retrieve_url function. Any thoughts?

Joe Schmid

02/07/2020, 7:59 PM
Hi @Nate Atkins, KilledWorker is often a result of a worker running out of memory or hitting a segfault. (Dask is preventing the task from running again because it has previously killed a worker.) This link has more details: Can you watch the Dask dashboard Workers tab for worker memory while your Flow runs? (Usually available on port 8787, e.g. http://localhost:8787 if you're running the Dask scheduler on your laptop.) Also see if there's any relevant output from the Dask workers.