emre

    emre

    1 year ago
    Hi everyone, I am migrating one of our core only runs to prefect server, but couldn’t get the tasks to work in parallel. The core only run uses
    flow.run(executor=LocalDaskExecutor(scheduler="threads", num_workers=4))
    and can execute tasks in parallel. The server registration is as follows:
    flow.storage = Docker(....)
    flow.run_config = DockerRun()
    flow.executor = LocalDaskExecutor(scheduler="threads", num_workers=4)
    flow.register(project_name=... , build=False)
    The flow can run and communicate with prefect server, but the tasks are executed sequentially. Both the gannt chart and the logs show this. Am I missing something? (prefect==0.13.13)
    v

    Vladimir Zoubritsky

    1 year ago
    Hello! We encountered something similar with the executor being ignored, but using
    environment
    instead worked
    flow.environment = LocalEnvironment(executor=LocalDaskExecutor(scheduler="threads", num_workers=4))
    emre

    emre

    1 year ago
    I had initially tried environments, exactly the same way, didn’t work. Thats when I found out environments are deprecated, and substituted for run_config + executor. I am suspecting that my executor choices aren’t being picked up by server, and prefect is falling back to the default
    LocalExecutor
    . Can you (or anyone else) verify whether prefect server shows information about the executor for a registered flow? Either directly under the flow details, or under the flow environment details?
    v

    Vladimir Zoubritsky

    1 year ago
    I did not see executor information in the flow details (or in the GraphQL responses) I've assigned the environment at the time of the flow initialization and not as a later assignment (and not the time of register), if that matters
    emre

    emre

    1 year ago
    Time of assignment was the issue 😅 . I was assigning server related members (
    flow.storage,flow.run_config, flow.executor
    ) at register time, while building a custom docker image, and serializing my flow in the docker image before register time, in a previous run of python. Therefore, my serialized flow pickle had empty values for
    flow.storage
    and
    flow.executor
    . I suppose only
    storage
    and
    run_config
    are sent to server, and
    executor
    is not. Since prefect server was not enforcing an executor on my flow run, and my serialized flow had no executor choice, I was falling back to the default
    LocalExecutor
    . Many thanks, discussing with you probably saved me multiple hours 😊
    Chris White

    Chris White

    1 year ago
    @Marvin archive “Registering a Flow with Prefect Server isn’t respecting my executor configuration”
    Marvin

    Marvin

    1 year ago