Hello Prefect team, Is there a way to log/monitor CPU and RAM usage of an entire Flow? Or maybe monitoring the usage of an Agent, or even better the total usage in the platform that is running the Agent?

@Marvin I'm trying to setup a deployment, I'm running into issue. I created a local process work pool named "local" but there's no worker available. When I

prefect worker start --pool "local"

I get:

ValueError: `PREFECT_API_URL` must be set to start a Worker.
An exception occurred.

@Marvin how to run prefect server in a docker container in nomd

Is it possible to run flows locally without posting results on the UI? (for testing)

I'm getting

Unable to start worker. Please ensure you have the necessary dependencies installed to run your desired worker type.

when I tried to start the Docker worker. Not sure if this is just misconfiguration of the pool or something missing on my machine?

Hello! I'm trying work my way through the docs on getting started with v.2.18.1 and I'm stalling out on the create a work flow piece. I'm able to authenticate via the CLI and move on to the next step but when I run this code:

from prefect import flow


@flow(log_prints=True)
def hello_world(name: str = "world", goodbye: bool = False):
    print(f"Hello {name} from Prefect! 🤗")

    if goodbye:
        print(f"Goodbye {name}!")


if __name__ == "__main__":
    hello_world.serve(name="my-first-deployment",
                      tags=["onboarding"],
                      parameters={"goodbye": True},
                      interval=60)

Then I get this message:

11:08:21.937 | WARNING | prefect.runner - Cannot schedule flows on an ephemeral server; run `prefect server start` to start the scheduler.
Your flow 'hello-world' is being served and polling for scheduled runs!

To trigger a run for this flow, use the following command:

        $ prefect deployment run 'hello-world/my-first-deployment'

It seems to run and the script continues to run but when checking my dashboard and trying to follow along in the steps, it never shows the run as actually scheduled. Any help would be appreciated.

Hi, I'm looking to set up some Prefect-agents-as-a-service within my organisation. We're planning to use the hosted Prefect Cloud with some AWS ECS agents, and a GitHub Actions pipeline to publish new flows (probably using S3 as storage). This is all greenfield so we're pretty flexible on the details. Right now we're sticking to Prefect 1.x but if there's a good reason to jump to 2.x I don't think that would be a problem (as long as it's safe/secure we don't mind the occasional hiccough while it's still in beta) Most of that is fine; so far we're just using a local agent deployed on EC2, but I can see there's an ECS agent which we can presumably use easily enough to get any scalability that we'll need, and we can use the

prefect

CLI to push flows from GitHub Actions. But where we're hitting problems is with dependency management (both internal code which is shared between multiple tasks/flows, and external dependencies). From what I've seen, Prefect doesn't really support this at all (flows are expected to be self-contained single files), with the implication being that the agent itself has to have any shared dependencies pre-installed (which in our case would mean that any significant changes require re-building and re-deploying the agent image - a slow process and not very practical if we have long-lived tasks or multiple people testing different flows at the same time). I tried looking around for Python bundlers and found stickytape, but that seems a bit too rough-and-ready for any real use. This seems to be a bit of a known problem: 1, 2 and specifically I see:

V2 supports virtual and conda environment specification per flow run which should help some

And I found some documentation for this (which seems to tie it to the new concept of deployments), but I'm still a bit confused on the details: • would the idea be to create a deployment for every version of every flow we push? Will we need to somehow tidy up the old deployments ourselves? • can deployments be given other internal files (i.e. common internal code), or is it limited to just external dependencies? Relatedly, do deployments live on the server or in the configured Storage? • is there any way to use zipapp bundles? • ideally we want engineers to be able to run flows in 3 ways: entirely locally; on a remote runner triggered from their local machine (with local code, including their latest local dependencies); and entirely remotely (pushed to the cloud server via an automated pipeline and triggered or scheduled - basically "push to production") — I'm not clear on how I should be thinking about deployments vs flows to make these 3 options a reality. I also wonder if I'm going down a complete rabbit hole and there is an easier way to do all of this?

Hello, I have a task that I am calling map on using Dask executor which works fine until there is an error. Sometimes I get an "Unexpected error: KilledWorker" which in my case seems to be because of dask worker has run out of memory. There are only a handful of cases where this happens but when it does it kills the whole flow run. Is there any way to handle this without killing the whole flow? I want to have 2 downstream tasks that would take the filtered results and act on them. Any help would be appreciated.

@Marvin What is the status of

DatabaseCredentials

? It seems like the github project for

prefect_sqlalchemy

is deprecated but the integration inside Prefect (

prefect[sqlalchemy]

) does not seems to support it. What is the proper way to read from a

Database Credentials

block? It it not marked as deprecated on the managed prefect (prefect.cloud)

@Marvin How do i await a prefectfuture without blocking the event loop?