Dumb question here... I am running a dask executor backend using coiled and using prefect cloud. It doesn't look like runs executing on the dask executor (with coiled) are being sent back to the prefect cloud.. I cannot see any runs, but the system works and runs everything correctly. Should I expect to be able to monitor runs from prefect cloud with this setup?

Hey @Brett Jurman, not a dumb question! Yes you should be able to monitor these. Could you show how you attach the exectur to your flow?

Here is my flow:

import prefect
from prefect import task, Flow
import time
import random

import numpy as np
from sklearn.linear_model import LinearRegression

@task
def pick_random_sizes():
    time.sleep(2*60)
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("Hello, Cloud!")

    return [random.randint(1e4, 1e6) for _ in range(10)]

@task
def train_model(size):
    m = LinearRegression()
    data = np.random.random((10000, 2))
    
    targets = np.random.random((10000, 1))
    
    m.fit(data, targets)
    
    time.sleep(60)
    
    return

@task
def eval_models(ms):
    for m in ms:
        print( m)
import prefect
from prefect import task, Flow
import time
import random

import numpy as np
from sklearn.linear_model import LinearRegression

@task
def pick_random_sizes():
    time.sleep(2*60)
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("Hello, Cloud!")

    return [random.randint(1e4, 1e6) for _ in range(10)]

@task
def train_model(size):
    m = LinearRegression()
    data = np.random.random((10000, 2))
    
    targets = np.random.random((10000, 1))
    
    m.fit(data, targets)
    
    time.sleep(60)
    
    return

@task
def eval_models(ms):
    for m in ms:
        print( m)

with Flow("full-flow") as flow:
    sizes = pick_random_sizes()
    
    models = train_model.map(sizes)
    
    eval_models(models)

# Register the flow under the "tutorial" project
flow.register(project_name="tester")
with Flow("full-flow") as flow:
    sizes = pick_random_sizes()
    
    models = train_model.map(sizes)
    
    eval_models(models)

# Register the flow under the "tutorial" project
flow.register(project_name="tester")

flow.run

won’t be on the UI. It’s meant for local testing. Flow runs of registered flows will be.

ah ok

When you register it, go to the UI and click the “Run Now”, it will then spin up the Coiled cluster because you registered with that executor. Flows that are registered (also the ones of schedule) will use the executor attached to the registration.

https://docs.coiled.io/user_guide/examples/prefect.html I am following the instructions here (the 2nd esxample)

There is a way to programatically kick off a flow using the client . You can do

client.create_flow_run

. All flow runs of registered flows will be tracked, just not the

flow.run

.

client.create_flow_run

will be though. The

agent

will need the Coiled key to spin up the executor. You need an agent running to run flows, so it’s that agent that will be authenticated to spin up the Coiled Executor. And not a problem!

wow thanks, gonna try that right now!

flow.register()

gives you an id that you can pass to the client. No you can’t spin up the executor without an agent. The agent is a long running process that listens for the flows and then starts them so there has to be something on that listens for flow runs. The Coiled cluster is ephemeral and only runs for the duration of the flow. The agent can be started with something like

prefect agent local start

. The agent is very lightweight (assuming all your flows outsource the compute) so a lot of users just use a minimal ec2 instance to host their agent that kicks off the flows

ah ok

Local agent will spin up the Coiled Executor and run the code there. Local will work for you.

ok great

I am yep