s

    Sergiy Krutsenko

    1 year ago
    Hi, I try to run Prefect Server on Docker box. All services are started and running and I can registry my flow and run from UI. But when it runs it uses LocalExceutor/ How can I make it to use Dask? I added DaskExecutor directly to the flow but it still runs with Local. I also tried to set env var PREFECT__ENGINE__EXECUTOR__DEFAULT_CLASS in docker-compose.yml file and in my flow but it did not help. How to make Prefect Server use Dask as executors while running flows? Update with code. Flow creation: def create_flow(name):   executor = get_executor(env='dev')   with Flow(name,           storage=Local(),           executor=executor) as flow:       input = Parameter('input_params', required=True)       data1 = _task1(input)       data2 = _taks2.map(data1)       data3 = _task3.map(data2)       data4 = _task4.map(data3)       result = _task5(data4)   return flow get_executor function returns DaskExecutor Flow registration: 
    flow = create_flow('TestFlowServer')
    flow.register(project_name="Test")
       docker-compose.yml Added below var to environment section for towel and graphql services PREFECT__ENGINE__EXECUTOR__DEFAULT_CLASS: prefect.engine.executors.DaskExecutor I use LocalAgent to run flows. I also added the above env var when I start it. But nothing helped. When I try to run the flow locally, without registering on Server everything works. Thanks, Sergey
    Dylan

    Dylan

    1 year ago
    Hi @Sergiy Krutsenko! Would you mind sharing your Flow code here? Specifically I’m interested in:1. Your Storage 2. Your environment / run config It would also be helpful to know your agent type (Docker, Local, Kubernetes, etc)
    s

    Sergiy Krutsenko

    1 year ago
    Hi @Dylan Here it is. Also edited my original post Flow creation:   def create_flow(name):   executor = get_executor(env='dev')   with Flow(name,           storage=Local(),           executor=executor) as flow:       input = Parameter('input_params', required=True)       data1 = _task1(input)       data2 = _taks2.map(data1)       data3 = _task3.map(data2)       data4 = _task4.map(data3)       result = _task5(data4)   return flow   get_executor function returns DaskExecutor   Flow registration: 
    flow = create_flow('TestFlowServer')
    flow.register(project_name="Test")
       docker-compose.yml   Added below var to environment section for towel and graphql services PREFECT__ENGINE__EXECUTOR__DEFAULT_CLASS: prefect.engine.executors.DaskExecutor   I use LocalAgent to run flows. I also added the above env var when I start it. But nothing helped.   When I try to run the flow locally, without registering on Server everything works.    Thanks, Sergiy
    Dylan

    Dylan

    1 year ago
    Would you mind sharing your
    get_executor
    function?
    Also, Prefect Server is the orchestration layer of the stack and isn’t responsible for how your flow runs, that’s defined on the flow object itself. So setting that environment variable in the Prefect Server services shouldn’t affect execution
    You may need to configure a
    LocalRun
    for this flow. Otherwise, I believe the Flow looks for a run config from the agent
    s

    Sergiy Krutsenko

    1 year ago
    @Dylan I fund another solution. I use RemoteDaskEnvironment with flow and it works now.
    Dylan

    Dylan

    1 year ago
    Glad to hear it!