https://prefect.io logo
a

Adam Lewis

03/17/2021, 1:52 PM
Hi everyone, I have prefect-server set up via minikube and the official helm charts (prefect-server-2021.03.10) including a Kubernetes Agent set up via helm chart. I want a flow to use the DaskExecutor so I've set the flow.executor to an instance of the DaskExecutor. I'm able to register and run the flow, but when I run the flow a job pod starts, but no other pods do where I would expect dask workers to spin up. Does anyone know what is going on here? The simple task and dask worker definitions can be seen at https://github.com/Adam-D-Lewis/sample-flow if needed.
j

Jim Crist-Harif

03/17/2021, 1:55 PM
Hi Adam, the issue here is that you're configuring your executor in a section of code that will only run if your flow source is run as a script (within a check like below)
Copy code
if __name__ == "__main__":
   ...
When prefect loads your flow to execute it, it is not run as a script, so that block won't execute and your executor won't be configured. I recommend only putting the
flow.register
call within that block and moving the rest of the configuration up above that check.
btw, thanks for providing a direct example of the issue you're having, this makes it so much easier to debug for us :)
a

Adam Lewis

03/17/2021, 1:57 PM
Ahh, I had assumed I only needed to set the DaskExecutor when registering the the flow. I'll give the changes you mention a try. Thanks for the quick response!
j

Jim Crist-Harif

03/17/2021, 1:58 PM
If you were using "pickle-based" storage that would be true, but github storage is "script-based", so your flow source is executed during every execution. See https://docs.prefect.io/orchestration/flow_config/storage.html#pickle-vs-script-based-storage for more info on the difference.
2 Views