Hi! I am new to using prefect but I am trying to u...
# prefect-server
Hi! I am new to using prefect but I am trying to utilize my dask cluster and having issues doing so on-prem. I ran into issues where the workers were looking to submit logs to
although the prefect server/agent were running on a dedicated scheduler box. Am I missing something for configuring the workers to work properly? Do they need to point back at the scheduler's ip for logging? Do I need to run an agent on each machine to pass jobs to the workers?
What error message do you see on the workers? How are you using the Dask cluster with Prefect? Are you passing the address to
Can you replicate with a simple Flow?
Copy code
import os
import time

import prefect
from prefect import task, Flow, Parameter
from prefect.run_configs import LocalRun
from prefect.executors import LocalDaskExecutor, DaskExecutor

def say_hello(name):
    # Add a sleep to simulate some long-running task
    # Load the greeting to use from an environment variable
    greeting = os.environ.get("GREETING")
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>(f"{greeting}, {name}!")

with Flow("hello-flow") as flow:
    people = Parameter("people", default=["Arthur", "Ford", "Marvin"])
    import construction

# Configure the `GREETING` environment variable for this flow
flow.run_config = LocalRun(env={"GREETING": "Hello"})

# Use a `LocalDaskExecutor` to run this flow
# This will run tasks in a thread pool, allowing for parallel execution
# flow.executor = LocalDaskExecutor()
flow.executor = DaskExecutor(address="<tcp://>")

# Register the flow under the "tutorial" project
flow.register(project_name="tutorial",  idempotency_key=flow.serialized_hash())
it outputs this in the terminals on the cluster workers
Copy code
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=4200): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x000002E6E10DAE20>: Failed to establish a new connection: [WinError 10061] No connection 
could be made because the target machine actively refused it'))
I see I think this is the client side as opposed to the workers. Does you agent live on the same box as where you started server?
Oh just saw you said you saw this on workers. I’ll look around, but I may have to find a team member who knows more and get. back to you tom
thank you!
the agent is on the same box as the server yes, but no agents on the worker boxes
This works for the LocalDaskExecutor?
I was trying it locally and it seemed to be okay, but when trying to attach the flow to the remote dask scheduler is when I ran into problems
This looks good as far as I can tell. I’ll ask team members who know more than me and get back to you tom.
awesome thank you! no rush
Hey @nick vazquez -- I think what you need to do here is set up the Prefect config on your dask workers so they point to your dedicated box
With Cloud, this generally isn't a concern since the endpoint is consistent across your agent/workers/etc but when using Server you need to set these endpoints
Ahh okay thank you!
sorry for the late response and thank you for your quick response! 🙂