Hi, is it possible to register a flow with a project, send it to a Dask cluster and keep track of it in Prefect GUI ?

Hi @Juan - yes: you'll need to have either a Prefect Server instance running or a Prefect Cloud account; I recommend the latter for ease of getting started since it's free and already set up. Then when registering your flow (which you've configured to run on Dask) with whichever you chose, you'll specify the project, which you can create through the UI. Once that's set up, all you need to do is run your flow! An expanded list of these steps can be found in the Deployment Tutorial.

Hi @nicholas, I already have a prefect server running. This is what I am doing:

flow = Flow("sklearn-iris-flow-dask", tasks=[load_data,fit_model,make_predictions,sumarize_fit_model])

flow.register(project_name="sklearn-iris")

from prefect.engine.executors import DaskExecutor

executor = DaskExecutor(address="<tcp://ip-scheduler>:port")

flow.run(executor=executor)

And I am getting this error:

[2020-09-11 12:36:45] INFO - prefect.FlowRunner | Beginning Flow run for 'sklearn-iris-flow-dask'
[2020-09-11 12:36:46] INFO - prefect.FlowRunner | Flow run FAILED: some reference tasks failed.

Any advice? Thanks !!

flow.run

will run your flow locally (with your given executor); since it doesn't look like any tasks started, I'd guess your Dask cluster isn't available. Please take a look at the links I posted above to run the flow from the UI.

Thanks for your answer. I removed the line

flow = Flow("sklearn-iris-flow", tasks=[load_data,fit_model,make_predictions,sumarize_fit_model])

and it worked. Although after some successful executions Prefect GUI shows all flows as scheduled even if I start a flow manually from GUI.

Hi @Juan - do you have an agent running that has all the labels your flow has?

Hi @nicholas. What I am doing is using python scripts to create tasks and register the flow on Prefect server. The Agent is running locally and executes the flow, something was wrong and the Agent was running but doing nothing.