Hi! Quick question. I'm running a prefect flow using the DaskExecutor, which is connected to a cluster. My checkpointing is turned on, but there is no result being save and no logs being printed out (DEBUG mode on). I get the results and the print-out if I use the local executor or a local instance of a DaskExecutor. Any ideas what to do in this case?

Hi Matic! Are you using

flow.run

or an orchestration backend for this?

Hi Chris! thanks for taking your time. I'm using a simple

flow.run

, yes.

Are you using a

LocalResult

for your flow? My first guess is that results are being written on the worker nodes, so the results would be local to the worker filesystems not your flow-runner system. Likewise, logs are written to stdout on the nodes the tasks are run, so your task logs are logged on the workers not local to the

flow.run

call.

For centralized logging on Dask, you can switch to using a Prefect API (which will collect all logs), or setting up another logging service that collects your worker logs

Hi Jim, thanks to you as well. Good insight! Indeed I am using the

LocalResult

and what you are describing here makes a lot of sense.

Definitely: https://docs.prefect.io/orchestration/ I recommend starting with Prefect Cloud (you still retain full control of your dask cluster, but your logs will be sent to the API for centralization) which you can sign up for (for free) here: https://cloud.prefect.io/

The tutorial is probably the best place to start: https://docs.prefect.io/orchestration/tutorial/overview.html

Thank you both very much for the super quick and helpful responses, I'll go through the documentation and tutorial first thing tomorrow. 🙏

Yep, that's totally fine! Agents determine where the flow runner process runs (main process for running a flow), while executors determine where tasks run (which may be in additional processes started somewhere else). This lets you mix-and-match agents and executors as needed. For example, you can use a local agent with a

DaskExecutor

configured to start a cluster on ECS. Or an ECS agent configured to use a local dask cluster (local meaning local to the same container as the flow runner).