Hi! Quick question. I'm running a prefect flow usi...
# prefect-community
m
Hi! Quick question. I'm running a prefect flow using the DaskExecutor, which is connected to a cluster. My checkpointing is turned on, but there is no result being save and no logs being printed out (DEBUG mode on). I get the results and the print-out if I use the local executor or a local instance of a DaskExecutor. Any ideas what to do in this case?
c
Hi Matic! Are you using
flow.run
or an orchestration backend for this?
m
Hi Chris! thanks for taking your time. I'm using a simple
flow.run
, yes.
👍 1
j
Are you using a
LocalResult
for your flow? My first guess is that results are being written on the worker nodes, so the results would be local to the worker filesystems not your flow-runner system. Likewise, logs are written to stdout on the nodes the tasks are run, so your task logs are logged on the workers not local to the
flow.run
call.
upvote 1
c
For centralized logging on Dask, you can switch to using a Prefect API (which will collect all logs), or setting up another logging service that collects your worker logs
m
Hi Jim, thanks to you as well. Good insight! Indeed I am using the
LocalResult
and what you are describing here makes a lot of sense.
Chris, the former sounds like the more elegant first thing to try. Can you point me to this API which you speak of, please?
c
Definitely: https://docs.prefect.io/orchestration/ I recommend starting with Prefect Cloud (you still retain full control of your dask cluster, but your logs will be sent to the API for centralization) which you can sign up for (for free) here: https://cloud.prefect.io/
j
The tutorial is probably the best place to start: https://docs.prefect.io/orchestration/tutorial/overview.html
upvote 1
m
Thank you both very much for the super quick and helpful responses, I'll go through the documentation and tutorial first thing tomorrow. 🙏
Thanks again, Chris and Jim! I managed to set up the server and use the PrefectResult, which points to a good direction, but haven't tried with a cluster yet. But before I proceed, I have another question. If I set the DaskExecutor to the flow and register the flow, is it OK if I use the local agent? I've stumbled upon the ECS agent but I can't wrap my head around it
j
Yep, that's totally fine! Agents determine where the flow runner process runs (main process for running a flow), while executors determine where tasks run (which may be in additional processes started somewhere else). This lets you mix-and-match agents and executors as needed. For example, you can use a local agent with a
DaskExecutor
configured to start a cluster on ECS. Or an ECS agent configured to use a local dask cluster (local meaning local to the same container as the flow runner).
upvote 1