I'm on v2.0b7 and am trying to get a minimal code example to work with a Dask cluster. I can use DaskTaskRunner locally, but as soon as I add the TCP address to my test cluster, the system fails. I see a very lengthy error on the Dask worker that includes: "prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url" I am able to submit jobs successfully using Dask directly, just not through Prefect. Here is the code:

from prefect import flow, task
from prefect_dask.task_runners import DaskTaskRunner

@task
def say_hello(name):
    print(f"hello {name}")

@flow(task_runner=DaskTaskRunner(address='tcp://[redacted]:8786'))
def greetings():
    say_hello('test')

if __name__ == "__main__":
    greetings()

Any ideas?

It sounds like your Dask worker cannot contact the Prefect API

Thanks for the quick response! The worker machine indeed has no internet connection. Makes perfect sense, addressing now.

Yeah, that looks like

PREFECT_API_URL

isn’t set on your worker at all.

How would I set the

PREFECT_API_URL

for a flow run? Here's what I've tried: • Go to submitter machine (the machine where I run the originally posted code) • Launch the Anaconda Prompt and activate an environment ("dask-demo") that has Dask and Prefect installed • Run

prefect orion start

in the Anaconda Prompt for the submitter machine • Note the

PREFECT_API_URL

that is printed after the Prefect logo in the console • Go to the machine that will be the dask-scheduler • Launch the Anaconda Prompt and activate "dask-demo" • Run

prefect config set PREFECT_API_URL=[the url]

where [the url] is the previously noted

PREFECT_API_URL

• Run

dask-scheduler

, note the TCP address • Go to the machine that will be the dask-worker • Launch the Anaconda Prompt and activate "dask-demo" • Run

prefect config set PREFECT_API_URL=[the url]

• Run

dask-worker [the tcp address]

• Go to the submitter machine and launch another instance of the Anaconda Prompt and activate "dask-demo" • Run

python [full path to my test script]

to run the originally posted code After following those steps I see "Crash detected!" and flow run "Finished in state Failed" in the prompt on the submitter machine. I also see

prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url '<http://ephemeral-orion/api/task_runs/.../set_state>

Thanks for the detailed description! The API URL is passed from the flow run to the task runs. So you’ll want to set the

PREFECT_API_URL

on the submitter machine. We’ll automatically set it within each task run when they are run on the Dask worker. If this URL needs to be different for the flow run and task run then I think you’ll run into problems as I don’t think we support splitting this yet.

Okay amazing, it looks like it's working! And great to know that only the submitter machine needs the

PREFECT_API_URL

. One assumption I made was that the submitter machine would already know that URL path since that's the machine where the

prefect orion start

was running. However, I now understand that since Prefect Orion and the python script were each running in separate conda prompt instances, that API URL was only known to the Prefect Orion conda prompt instance. I went ahead and removed the API URL info from all of the dask machines and confirmed that indeed all I need to do is run

prefect config set PREFECT_API_URL=[the url]

and then run the python script from that same conda prompt instance. Thank you again so much for the help!

Wonderful! cc @terrence we’ll need documentation on this pattern in the future.