I'm trying to run a flow using LocalEnvironment with DaskExecutor but the flow run never starts. It just keeps logging that the flow has entered the running state, and the Dask dashboard shows there's nothing running on the cluster. Has anyone ran into this problem before?

flow.environment = LocalEnvironment(
            executor=DaskExecutor(
                adapt_kwargs={
                    "minimum": 1,
                    "maximum": 5,
                },
            ),
        )

FWIW, I'm using prefect 0.13.2, dask 2.26.0, distributed 2.26.0

Hi @Lewis Bails, that certainly is odd, I've never seen that issue before. Two things I'd try: • Does your flow run fine locally doing

flow.run(executor=...)

with the same executor configured above? • Does your flow run fine with a local environment configured with a

DaskExecutor

(without the

adapt_kwargs

)? This will create a local cluster based on the amount of cores available to your local run. If this works, there's a bug in adaptive scaling for the local cluster (unlikely, but possible).

Hey @Jim Crist-Harif, it works like a charm if I run it locally. No luck getting it going with the Agent after removing the DaskExecutor arguments.

No luck getting it going with the Agent after removing the DaskExecutor arguments.

Just to check, did you re-register the flow after making that change (to make sure you were using the updated code)? --- A few other questions: • Can you successfully run any flow on the agent (one that doesn't use dask at all?) • What agent are you using? If you're using the local agent, can you use

--show-flow-logs

when starting the agent to get stdout from the flow runs? This will let you see all stdout output from prefect, not just log messages (dask logs generally aren't picked up by prefect, but will aid in debugging). Using the following executor would also give you more log output:

executor = DaskExecutor(cluster_kwargs={"silence_logs": 10})

Yeah re-registered and the version number got bumped on the UI so I assume that worked. My other tasks that just run with the default executor run perfectly fine. You're right in that it's just a local agent, I'll setup the logs stuff and get back to you!