how do you verify that a DaskExecutor was used to run a flow? There is no info about the executor in...

xyzz

03/30/2021, 2:44 PM

how do you verify that a DaskExecutor was used to run a flow? There is no info about the executor in the logs, right?

xyzz

03/30/2021, 2:46 PM

Reason I'm asking is that I'd like to only add the DaskExecutor when registering the flow in a separate deploy.py, not in the flow.py itself. But I'm not sure if this is supported

xyzz

03/30/2021, 2:47 PM

For RunConfig this approach seems to work, but for Storage it doesn't

Mariia Kerimova

03/30/2021, 4:43 PM

Hello Klaus! We do log which executor is used, but it's log of Debug level (we should change it to Info though).

xyzy

03/30/2021, 5:13 PM

Thats strange, my the header on the log says "Showing logs for all log levels from the beginning of time to now.". But I don't see anything about the executor, only the storage is mentioned

🤔 1

Mariia Kerimova

03/30/2021, 6:08 PM

You might found a bug, but before opening the issue. can I ask you to try the following: • in

~/.prefect/config.toml

set:

Copy code

[logging]
level = "DEBUG"

• run your agent with

prefect agent <type> <-t token if needed> --show-flow-logs

• check if agent logs have Debug logs or sentence

Using executor type

xyzy

03/30/2021, 7:38 PM

I'm trying to get this running with a local docker for windows kubernetes cluster (just for dev), but after uninstalling the helm release and trying to reinstall it with "--set prefectConfig.logging.level=Debug" option my graphql pod is in state Init:CrashLoopBackOff

xyzy

03/30/2021, 7:38 PM

the logs only show the following: "Error from server (BadRequest): container "graphql" in pod "prefect-server-graphql-84f4d57c79-blg68" is waiting to start: PodInitial izing"

xyzy

03/30/2021, 7:43 PM

the last two events in describe pod are:

Copy code

Normal   Pulled     6m18s                   kubelet            Successfully pulled image "prefecthq/server:2021.03.20" in 1.370319s
  Warning  BackOff    2m38s (x25 over 7m38s)  kubelet            Back-off restarting failed container

xyzy

03/30/2021, 7:47 PM

figured it out... i passed Debug but it need to be DEBUG

xyzy

03/30/2021, 7:52 PM

...but it seems this isn't passed to the agent that is created with helm option --set agent.enabled=true? I still don't get any debug lines in the server gui

xyzy

03/30/2021, 7:53 PM

how can I tell the agent integrated in the helm chart to set log level to DEBUG?

xyzy

03/30/2021, 8:01 PM

For now I've tested with a local agent... the Using executor type line shows up with logging level = DEBUG

xyzy

03/30/2021, 8:02 PM

Seems like the Executor type has to be set in the flow code that is executed on the pod... I'd have hoped it would be enough to set it when registering the flow to keep the code DRY

xyzy

03/30/2021, 8:02 PM

since I want to use the default executor when debugging locally

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.