https://prefect.io logo
#prefect-community
Title
# prefect-community
n

Nate Atkins

02/07/2020, 7:59 PM
This runs with line 8 commented out, but raises the KilledWorker when it is run.
j

Joe Schmid

02/07/2020, 8:03 PM
@Nate Atkins Maybe your Dask workers don't have the
requests
module installed? Make sure to start them with a python environment/virtualenv that has
requests
installed.
n

Nate Atkins

02/07/2020, 8:07 PM
I thought about that. I checked to make sure I was starting the workers in the venv and directory that the working example runs in when I let the executor start the cluster. It also seems like commenting out the requests.get() would still blow up as the model is still loaded. Maybe this isn't the case as I don't fully understand what is getting passed from the scheduler to the worker. I can check this by removing the package and see what happens.
This is what I see in the dask-worker output. [2020-02-07 201343,099] INFO - prefect.TaskRunner | Task 'retrieve_url': Starting task run... objc[68582]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. objc[68582]: +[__NSCFConstantString initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug. distributed.nanny - INFO - Worker process 68582 was killed by signal 6 distributed.nanny - WARNING - Restarting worker
Got it! https://github.com/darkskyapp/forecast-ruby/issues/13 On MacOS Mojave export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
j

Joe Schmid

02/07/2020, 8:21 PM
@Nate Atkins Nice, great find!
c

Chris White

02/07/2020, 9:23 PM
@Marvin archive “Unexpected error occured in FlowRunner: KilledWorker”