I've been working through the tutorial and ran into a Dask KilledWorker error and was hoping someone might be able to point me to a solution or give me some pointers on how to debug this.
I can start dask-scheduler and 2 dask-workers to successfully run the "Deployment: Dask" example.
https://docs.prefect.io/core/tutorials/dask-cluster.html
I can run the "Advanced Features" example through the DaskExecutor section where the DaskExecutor starts up the dask-cluster and process the URLs in 20% of the single threaded mode.
https://docs.prefect.io/core/tutorials/advanced-mapping.html#outline
If I add the scheduler address to the executor parameters
executor = DaskExecutor(address="<tcp://192.168.86.23:8786>")
flow.run(executor=executor)
I get the KilledWorker error.
ERROR - prefect.Flow: x-files | Unexpected error occured in FlowRunner: KilledWorker('retrieve_url-a032a3d8-fa89-4131-ad4c-4f3c5fbd282c', <Worker '
tcp://192.168.86.23:58923', name:
tcp://192.168.86.23:58923, memory: 0, processing: 1>)
By commenting things out I'm down to the offending line being
html = requests.get(url)
in the retrieve_url function.
Any thoughts?