Hi! I am new to using prefect but I am trying to utilize my dask cluster and having issues doing so on-prem. I ran into issues where the workers were looking to submit logs to

localhost:4200

although the prefect server/agent were running on a dedicated scheduler box. Am I missing something for configuring the workers to work properly? Do they need to point back at the scheduler's ip for logging? Do I need to run an agent on each machine to pass jobs to the workers?

What error message do you see on the workers? How are you using the Dask cluster with Prefect? Are you passing the address to

DaskExecutor

?

from

import os
import time

import prefect
from prefect import task, Flow, Parameter
from prefect.run_configs import LocalRun
from prefect.executors import LocalDaskExecutor, DaskExecutor


@task
def say_hello(name):
    # Add a sleep to simulate some long-running task
    time.sleep(10)
    # Load the greeting to use from an environment variable
    greeting = os.environ.get("GREETING")
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>(f"{greeting}, {name}!")


with Flow("hello-flow") as flow:
    people = Parameter("people", default=["Arthur", "Ford", "Marvin"])
    say_hello.map(people)
    
    import construction

# Configure the `GREETING` environment variable for this flow
flow.run_config = LocalRun(env={"GREETING": "Hello"})

# Use a `LocalDaskExecutor` to run this flow
# This will run tasks in a thread pool, allowing for parallel execution
# flow.executor = LocalDaskExecutor()
flow.executor = DaskExecutor(address="<tcp://192.168.168.114:8786>")


# Register the flow under the "tutorial" project
flow.register(project_name="tutorial",  idempotency_key=flow.serialized_hash())

it outputs this in the terminals on the cluster workers

requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000002E6E10DAE20>: Failed to establish a new connection: [WinError 10061] No connection 
could be made because the target machine actively refused it'))

I see I think this is the client side as opposed to the workers. Does you agent live on the same box as where you started server?

thank you!

This works for the LocalDaskExecutor?

I was trying it locally and it seemed to be okay, but when trying to attach the flow to the remote dask scheduler is when I ran into problems

This looks good as far as I can tell. I’ll ask team members who know more than me and get back to you tom.

awesome thank you! no rush

Hey @nick vazquez -- I think what you need to do here is set up the Prefect config on your dask workers so they point to your dedicated box

Ahh okay thank you!