https://prefect.io logo
m

Matic Lubej

01/20/2021, 9:49 PM
Hi! Quick question. I'm running a prefect flow using the DaskExecutor, which is connected to a cluster. My checkpointing is turned on, but there is no result being save and no logs being printed out (DEBUG mode on). I get the results and the print-out if I use the local executor or a local instance of a DaskExecutor. Any ideas what to do in this case?
c

Chris White

01/20/2021, 9:50 PM
Hi Matic! Are you using
flow.run
or an orchestration backend for this?
m

Matic Lubej

01/20/2021, 9:50 PM
Hi Chris! thanks for taking your time. I'm using a simple
flow.run
, yes.
👍 1
j

Jim Crist-Harif

01/20/2021, 9:51 PM
Are you using a
LocalResult
for your flow? My first guess is that results are being written on the worker nodes, so the results would be local to the worker filesystems not your flow-runner system. Likewise, logs are written to stdout on the nodes the tasks are run, so your task logs are logged on the workers not local to the
flow.run
call.
upvote 1
c

Chris White

01/20/2021, 9:53 PM
For centralized logging on Dask, you can switch to using a Prefect API (which will collect all logs), or setting up another logging service that collects your worker logs
m

Matic Lubej

01/20/2021, 9:54 PM
Hi Jim, thanks to you as well. Good insight! Indeed I am using the
LocalResult
and what you are describing here makes a lot of sense.
Chris, the former sounds like the more elegant first thing to try. Can you point me to this API which you speak of, please?
c

Chris White

01/20/2021, 9:57 PM
Definitely: https://docs.prefect.io/orchestration/ I recommend starting with Prefect Cloud (you still retain full control of your dask cluster, but your logs will be sent to the API for centralization) which you can sign up for (for free) here: https://cloud.prefect.io/
j

Jim Crist-Harif

01/20/2021, 9:57 PM
The tutorial is probably the best place to start: https://docs.prefect.io/orchestration/tutorial/overview.html
upvote 1
m

Matic Lubej

01/20/2021, 10:01 PM
Thank you both very much for the super quick and helpful responses, I'll go through the documentation and tutorial first thing tomorrow. 🙏
Thanks again, Chris and Jim! I managed to set up the server and use the PrefectResult, which points to a good direction, but haven't tried with a cluster yet. But before I proceed, I have another question. If I set the DaskExecutor to the flow and register the flow, is it OK if I use the local agent? I've stumbled upon the ECS agent but I can't wrap my head around it
j

Jim Crist-Harif

01/21/2021, 2:40 PM
Yep, that's totally fine! Agents determine where the flow runner process runs (main process for running a flow), while executors determine where tasks run (which may be in additional processes started somewhere else). This lets you mix-and-match agents and executors as needed. For example, you can use a local agent with a
DaskExecutor
configured to start a cluster on ECS. Or an ECS agent configured to use a local dask cluster (local meaning local to the same container as the flow runner).
upvote 1