https://prefect.io logo
Title
b

Brett Jurman

07/13/2021, 3:42 PM
Dumb question here... I am running a dask executor backend using coiled and using prefect cloud. It doesn't look like runs executing on the dask executor (with coiled) are being sent back to the prefect cloud.. I cannot see any runs, but the system works and runs everything correctly. Should I expect to be able to monitor runs from prefect cloud with this setup?
k

Kevin Kho

07/13/2021, 3:44 PM
Hey @Brett Jurman, not a dumb question! Yes you should be able to monitor these. Could you show how you attach the exectur to your flow?
b

Brett Jurman

07/13/2021, 3:45 PM
Here is my flow:
import prefect
from prefect import task, Flow
import time
import random

import numpy as np
from sklearn.linear_model import LinearRegression

@task
def pick_random_sizes():
    time.sleep(2*60)
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("Hello, Cloud!")

    return [random.randint(1e4, 1e6) for _ in range(10)]

@task
def train_model(size):
    m = LinearRegression()
    data = np.random.random((10000, 2))
    
    targets = np.random.random((10000, 1))
    
    m.fit(data, targets)
    
    time.sleep(60)
    
    return

@task
def eval_models(ms):
    for m in ms:
        print( m)
import prefect
from prefect import task, Flow
import time
import random

import numpy as np
from sklearn.linear_model import LinearRegression

@task
def pick_random_sizes():
    time.sleep(2*60)
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("Hello, Cloud!")

    return [random.randint(1e4, 1e6) for _ in range(10)]

@task
def train_model(size):
    m = LinearRegression()
    data = np.random.random((10000, 2))
    
    targets = np.random.random((10000, 1))
    
    m.fit(data, targets)
    
    time.sleep(60)
    
    return

@task
def eval_models(ms):
    for m in ms:
        print( m)

with Flow("full-flow") as flow:
    sizes = pick_random_sizes()
    
    models = train_model.map(sizes)
    
    eval_models(models)

# Register the flow under the "tutorial" project
flow.register(project_name="tester")
with Flow("full-flow") as flow:
    sizes = pick_random_sizes()
    
    models = train_model.map(sizes)
    
    eval_models(models)

# Register the flow under the "tutorial" project
flow.register(project_name="tester")
and here is how im running it
executor = DaskExecutor(
    cluster_class=coiled.Cluster,
    cluster_kwargs={
        "software": "my-software-env",
        "shutdown_on_close": True,
        "name": "prefect-executor",
    },
)

flow.run(
    executor=executor,
)
im currently evaluating the framework, obviously this is not a real world flow 🙂
its running nicely on the dask cluster in coiled, i just dont ssee nayhitng in the ui of prefect
k

Kevin Kho

07/13/2021, 3:48 PM
flow.run
won’t be on the UI. It’s meant for local testing. Flow runs of registered flows will be.
b

Brett Jurman

07/13/2021, 3:49 PM
ah ok
is ther ea way to run a registered flow on the dask cluster in coiled?
i can confirm that coiled is the thing running hte cluster
k

Kevin Kho

07/13/2021, 3:50 PM
When you register it, go to the UI and click the “Run Now”, it will then spin up the Coiled cluster because you registered with that executor. Flows that are registered (also the ones of schedule) will use the executor attached to the registration.
b

Brett Jurman

07/13/2021, 3:50 PM
https://docs.coiled.io/user_guide/examples/prefect.html I am following the instructions here (the 2nd esxample)
oh ok
is there a way to programatically kick off a flow
and stillhave it tracked by the ui?
how would that work though, because wouldnt the executor need to know my coiled key.. that should be only on my local machine right?
sorry to pester you on this?
k

Kevin Kho

07/13/2021, 3:56 PM
There is a way to programatically kick off a flow using the client . You can do
client.create_flow_run
. All flow runs of registered flows will be tracked, just not the
flow.run
.
client.create_flow_run
will be though. The
agent
will need the Coiled key to spin up the executor. You need an agent running to run flows, so it’s that agent that will be authenticated to spin up the Coiled Executor. And not a problem!
b

Brett Jurman

07/13/2021, 3:58 PM
wow thanks, gonna try that right now!
i think thats the confusing part though
the agent itself as i understand is running on coiled
i believe i should be able to run a coiled exectuor without having to spin up my own agents
k

Kevin Kho

07/13/2021, 4:04 PM
flow.register()
gives you an id that you can pass to the client. No you can’t spin up the executor without an agent. The agent is a long running process that listens for the flows and then starts them so there has to be something on that listens for flow runs. The Coiled cluster is ephemeral and only runs for the duration of the flow. The agent can be started with something like
prefect agent local start
. The agent is very lightweight (assuming all your flows outsource the compute) so a lot of users just use a minimal ec2 instance to host their agent that kicks off the flows
b

Brett Jurman

07/13/2021, 4:05 PM
ah ok
so the agent will somehow run the correct exectuor
ok got it
and i can just run that locally
ok so the last thing is
docker      Manage Prefect Docker agents.
  ecs         Manage Prefect ECS agents.
  fargate     Manage Prefect Fargate agents (DEPRECATED).
  kubernetes  Manage Prefect Kubernetes agents.
  local       Manage Prefect Local agents.
these are my optioins for running prefect agent, should I choose local? the dask / coiled option is not clearly here
k

Kevin Kho

07/13/2021, 4:07 PM
Local agent will spin up the Coiled Executor and run the code there. Local will work for you.
b

Brett Jurman

07/13/2021, 4:07 PM
ok great
this is everything i needed
👍 1
are you a prefect employee? im trying it now and it seesm to b eworking
k

Kevin Kho

07/13/2021, 4:08 PM
I am yep