https://prefect.io logo
Title
j

Juan

09/11/2020, 1:50 PM
Hi, is it possible to register a flow with a project, send it to a Dask cluster and keep track of it in Prefect GUI ?
n

nicholas

09/11/2020, 2:13 PM
Hi @Juan - yes: you'll need to have either a Prefect Server instance running or a Prefect Cloud account; I recommend the latter for ease of getting started since it's free and already set up. Then when registering your flow (which you've configured to run on Dask) with whichever you chose, you'll specify the project, which you can create through the UI. Once that's set up, all you need to do is run your flow! An expanded list of these steps can be found in the Deployment Tutorial.
j

Juan

09/11/2020, 3:26 PM
Hi @nicholas, I already have a prefect server running. This is what I am doing:
flow = Flow("sklearn-iris-flow-dask", tasks=[load_data,fit_model,make_predictions,sumarize_fit_model])
flow.register(project_name="sklearn-iris")
from prefect.engine.executors import DaskExecutor
executor = DaskExecutor(address="<tcp://ip-scheduler>:port")
flow.run(executor=executor)
And I am getting this error:
[2020-09-11 12:36:45] INFO - prefect.FlowRunner | Beginning Flow run for 'sklearn-iris-flow-dask'
[2020-09-11 12:36:46] INFO - prefect.FlowRunner | Flow run FAILED: some reference tasks failed.
Any advice? Thanks !!
n

nicholas

09/11/2020, 3:31 PM
flow.run
will run your flow locally (with your given executor); since it doesn't look like any tasks started, I'd guess your Dask cluster isn't available. Please take a look at the links I posted above to run the flow from the UI.
j

Juan

09/14/2020, 7:11 AM
Thanks for your answer. I removed the line
flow = Flow("sklearn-iris-flow", tasks=[load_data,fit_model,make_predictions,sumarize_fit_model])
and it worked. Although after some successful executions Prefect GUI shows all flows as scheduled even if I start a flow manually from GUI.
n

nicholas

09/14/2020, 1:23 PM
Hi @Juan - do you have an agent running that has all the labels your flow has?
j

Juan

09/14/2020, 1:31 PM
Hi @nicholas. What I am doing is using python scripts to create tasks and register the flow on Prefect server. The Agent is running locally and executes the flow, something was wrong and the Agent was running but doing nothing.