https://prefect.io logo
l

Lewis Bails

09/22/2020, 9:54 AM
I'm trying to run a flow using LocalEnvironment with DaskExecutor but the flow run never starts. It just keeps logging that the flow has entered the running state, and the Dask dashboard shows there's nothing running on the cluster. Has anyone ran into this problem before?
Copy code
flow.environment = LocalEnvironment(
            executor=DaskExecutor(
                adapt_kwargs={
                    "minimum": 1,
                    "maximum": 5,
                },
            ),
        )
FWIW, I'm using prefect 0.13.2, dask 2.26.0, distributed 2.26.0
j

Jim Crist-Harif

09/22/2020, 1:56 PM
Hi @Lewis Bails, that certainly is odd, I've never seen that issue before. Two things I'd try: • Does your flow run fine locally doing
flow.run(executor=...)
with the same executor configured above? • Does your flow run fine with a local environment configured with a
DaskExecutor
(without the
adapt_kwargs
)? This will create a local cluster based on the amount of cores available to your local run. If this works, there's a bug in adaptive scaling for the local cluster (unlikely, but possible).
l

Lewis Bails

09/24/2020, 8:27 AM
Hey @Jim Crist-Harif, it works like a charm if I run it locally. No luck getting it going with the Agent after removing the DaskExecutor arguments.
j

Jim Crist-Harif

09/24/2020, 1:37 PM
No luck getting it going with the Agent after removing the DaskExecutor arguments.
Just to check, did you re-register the flow after making that change (to make sure you were using the updated code)? --- A few other questions: • Can you successfully run any flow on the agent (one that doesn't use dask at all?) • What agent are you using? If you're using the local agent, can you use
--show-flow-logs
when starting the agent to get stdout from the flow runs? This will let you see all stdout output from prefect, not just log messages (dask logs generally aren't picked up by prefect, but will aid in debugging). Using the following executor would also give you more log output:
Copy code
executor = DaskExecutor(cluster_kwargs={"silence_logs": 10})
l

Lewis Bails

09/25/2020, 5:48 AM
Yeah re-registered and the version number got bumped on the UI so I assume that worked. My other tasks that just run with the default executor run perfectly fine. You're right in that it's just a local agent, I'll setup the logs stuff and get back to you!
Thanks for the tip on the logs @Jim Crist-Harif, I could instantly see there was something happening with file permissions and dask-worker-space (I guess that's why running it locally worked), which led to it getting stuck. Silly error, all fixed now I think, thanks for your help!