Hi everyone, I am migrating one of our core only r...
# prefect-community
Hi everyone, I am migrating one of our core only runs to prefect server, but couldn’t get the tasks to work in parallel. The core only run uses
flow.run(executor=LocalDaskExecutor(scheduler="threads", num_workers=4))
and can execute tasks in parallel. The server registration is as follows:
Copy code
flow.storage = Docker(....)
flow.run_config = DockerRun()
flow.executor = LocalDaskExecutor(scheduler="threads", num_workers=4)
flow.register(project_name=... , build=False)
The flow can run and communicate with prefect server, but the tasks are executed sequentially. Both the gannt chart and the logs show this. Am I missing something? (prefect==0.13.13)
Hello! We encountered something similar with the executor being ignored, but using
instead worked
flow.environment = LocalEnvironment(executor=LocalDaskExecutor(scheduler="threads", num_workers=4))
upvote 2
I had initially tried environments, exactly the same way, didn’t work. Thats when I found out environments are deprecated, and substituted for run_config + executor. I am suspecting that my executor choices aren’t being picked up by server, and prefect is falling back to the default
. Can you (or anyone else) verify whether prefect server shows information about the executor for a registered flow? Either directly under the flow details, or under the flow environment details?
I did not see executor information in the flow details (or in the GraphQL responses) I've assigned the environment at the time of the flow initialization and not as a later assignment (and not the time of register), if that matters
Time of assignment was the issue 😅 . I was assigning server related members (
flow.storage,flow.run_config, flow.executor
) at register time, while building a custom docker image, and serializing my flow in the docker image before register time, in a previous run of python. Therefore, my serialized flow pickle had empty values for
. I suppose only
are sent to server, and
is not. Since prefect server was not enforcing an executor on my flow run, and my serialized flow had no executor choice, I was falling back to the default
. Many thanks, discussing with you probably saved me multiple hours 😊
👍 2
@Marvin archive “Registering a Flow with Prefect Server isn’t respecting my executor configuration”