Trying to test out prefect + coiled with gpus. i have a model training as a task and got "No heartbeat detected from the remote task; marking the run as failed." Is this something that can happen if the worker becomes too bogged down resource wise?

Hi all, We have multiple teams set up within our prefect account for different teams. Is it possible to share agents between teams?

Realistically speaking, how many people does it take to run and maintain prefect server? Is it like 0.5 people, 2 people?

How does the scheduling component of Prefect actually work? I get that the agent asks for work, but how does the API know which flows should be ran? I am guessing it has the next N scheduled times computed, but is that pre computed on some schedule? Or calculated on API request?

Hey everyone, I'm using the ECS agent and want to pass a secret from AWS Secrets Manager into each task. I saw the documentation about providing a task template for each task to build upon. do I have to create a fully defined template for each task? or can I just create the secrets{} part and let prefect auto generate the rest of it with defaults?

Hello there, I would like to ask how to shut down all process of my 'prefect server start'. I have found 1. https://github.com/PrefectHQ/prefect/issues/2348 2. https://github.com/PrefectHQ/prefect/issues/3660 But I can't find any documentation associated with prefect server stop Could you show me how to perform a docker-compose down similar behavior. Best reagards

Is there a utility, graphql mutation, or canned SQL query available for cleaning up the postgres database? I have a kubernetes deployment and I want to regularly clean up some of the larger tables (logs, task runs) while leaving recent data from the past 30 days intact so users don't suddenly lose all of their recent logs, tasks, etc.

Is there a way to set the api_token in code rather than relying on env or config file?

We have some Prefect flows that sometimes need to run for many hours. For particularly long-running flows, once the flow has been running for more than 12 hours, we’re often seeing that flow fail before it completes -- the Prefect UI shows its last state message as “`Unexpected error: CancelledError()`“. This doesn’t happen as a result of the code we’ve written to launch or monitor flows. It appears to be a result of an action that Prefect (or Dask?) is taking to automatically cancel long-running flows. However, I don’t see anything in the Prefect or Dask docs indicating that this is expected behavior, or how it could be controlled (e.g., disabled, or increased the allowable duration, etc.). Can anybody provide any guidance on how to deal with flows failing with this

CancelledError

? Any clues on how we can configure Prefect or Dask to allow flows to run past the 12 hour mark?

If I have a flow run in scheduled state with one parameter, then i try to run that flow again with a different param, why does it cancel the first flow run?

Hi Team, about the the storage options, can I use any DB for it, such as Postgres? If it’s not supported so far, is there any guide I can reference to and implement it by myself? Thanks

Hi - I’ve been going through the Prefect tutorials using a free account. I’ve gotten the basic Hello World flow to run using the local agent, but now I’m trying to get it to run via the kubernetes agent (using MacOS desktop docker). I’ve run the command-line k8s agent as follows:

prefect agent kubernetes start

. When I kick off the hello world flow, this indeed starts a job container on my desktop k8s which appears to start up fine and have the values that I configured (eg correct image). However it stays in a “running” state for several minutes until it errors out as follows:

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: /graphql (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f5455f07710>: Failed to establish a new connection: [Errno -3] Tempor

Here’s the code for my flow:

import prefect
from prefect import task, Flow
from prefect.run_configs import KubernetesRun

@task
def hello_task():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>("Hello world!")

flow = Flow("hello-flow2", tasks=[hello_task])

# flow.run() We could run our flow locally using the flow's run method but we'll be running this from Cloud!

flow.run_config = KubernetesRun(
    env={"PREFECT__LOGGING__LEVEL": "DEBUG"},
    image="prefecthq/prefect:latest",
    labels=[]
)
flow.register(project_name="tester")

Morning 👋 I’m using

DaskExecutor

with a custom

cluster_class

the logs in Prefect UI show me;

15:36:12 INFO DaskExecutor Creating a new Dask cluster with `dask_jobqueue.remote_slurm.RemoteSLURMCluster.with_http_api_client`...
15:36:13 INFO DaskExecutor The Dask dashboard is available at <http://10.240.3.45:8787/status>

But I know for a fact that RemoteSlurmCluster (by dask-jobqueue) also has logger.debug statement. How do I make those logger statements show up in either the flow logs (in the agent) or in the prefect logger UI?

Hey guys! I'm really stuck here, but sure i'm missing something easy within the documentation. I have a prefect server running on an independant linux machine and developing on a windows environment. I want to deploy my flow (from my windows environment) to my prefect server. Local storage doesn't seem to be an option as my linux machine has no access to the network that my .py file is on. So... git storage? We use Azure Devops as our repo but i'm honestly lost at how I can now deploy or connect that repo to my server. Any help would be greatly appreciated.

Hello, I have reached this point in the prefect documentation and I was wondering where I should be looking to be able to host my dashboard in the cloud instead of from my local server? https://docs.prefect.io/orchestration/server/deploy-local.html#ui-configuration

Found a potential difficulty/confusion with

prefect agent docker start --network host

where the network is added to the container but not able to use it as the primary interface to find

PREFECT__SERVER__HOST

behind a private DNS. https://github.com/PrefectHQ/prefect/issues/4840 Posting here for visibility

Sometimes with my coiled flows it takes long enough that prefect times out while starting the cluster. Is there a way to extend the timeout?

So ran into something new today after updating to 0.15.3 on all server components. For simplicity, here is a flow run creation in interactive UI:

mutation {
  create_flow_run(
    input: {
      flow_id: "f667e3f3-b5eb-4c52-8f90-36bc54633ac9", 
      parameters: "{\"file_id\": \"42556bd0-f495-11eb-a7a0-8c8590acd4d8\"}"
    }
  ) {
    id
  }
}

Which gives this output. We are getting the same output from our code that runs against graphql endpoint to trigger a flow run and has been working fine for months on 0.14.15.

{
  "errors": [
    {
      "message": "[{'extensions': {'path': '$.variableValues.insert_on_conflict.constraint', 'code': 'validation-failed'}, 'message': 'unexpected value \"ix_flow_run_idempotency_key_unique\" for enum: \\'flow_run_constraint\\''}]",
      "locations": [
        {
          "line": 2,
          "column": 3
        }
      ],
      "path": [
        "create_flow_run"
      ],
      "extensions": {
        "code": "INTERNAL_SERVER_ERROR",
        "exception": {
          "message": "[{'extensions': {'path': '$.variableValues.insert_on_conflict.constraint', 'code': 'validation-failed'}, 'message': 'unexpected value \"ix_flow_run_idempotency_key_unique\" for enum: \\'flow_run_constraint\\''}]"
        }
      }
    }
  ],
  "data": {
    "create_flow_run": null
  }
}

Hey guys! Glad to have found this community and hope everyone has had a good week so far! I've been trying to deploy the Prefect server in its own namespace on a shared EKS cluster - the cluster itself has 3 nodes and exists inside a VPC. I'd like to have the UI hosted in the cluster if possible, and have been trying to use DNS to connect it to the apollo endpoint without much luck (it just seems unable to connect at all). I know we have a DNS server up, though it's in a different namespace. This may be more of a k8s/EKS issue, but I figured I'd ask in case anyone else had run into it.

Hi! Too often something happens and flow freezes influentially. Flow should finish in few seconds, sometimes it work but sometimes it freezes forever. It is impossible to cancel it too. How to debug such problem? 10% of flow runs have such behaviour and it happens only if server side run in docker

does anyone know of a resource that will get me started with installing prefect server and an agent on an ubuntu VM? All i seem to be able to find is docs on GCP and AWS. I want it to run as a service too, restarting automatically if it crashes out

I have a very generic question about the Prefect architecture, the

Storage

in particular. My current project is using simple

flow.run()

s, all packaged in a Docker container. When switching to Prefect Server for orchestration, I have to register flows and thus define a

Storage

class,

RunConfig

etc. It seems like I have to write a lot of “infrastructure” code into my repo and reference different systems etc. I am not sure how to properly encapsulate this / separate these concerns from my business logic. I just want to be able to keep developing locally and use

flow.run

, but then have all the needed registration, storage configuration etc taken care of during deployment. I tried by simply adding a separate file in which I import the

flow

objects and then configure them before I register them, but e.g. with

Local()

storage and

save_as_script=True

, I then need to provide a path to the original flow file so I am not sure all the flow config made in the separate file will be taken into account. Long story short: I find the step from using core to orchestration very difficult to understand and to structure in terms of code - I have the feeling it’s kind of either-or. Positive closing: my experience with Prefect has been amazing so far 🙂

Hey everyone! I am new to Prefect and am loving it so far! I have deployed Prefect Server + Agent alongside a persistent Dask cluster in K8s, using Git storage for the following flow:

from prefect import task, Flow
import random
from time import sleep
from prefect.run_configs import KubernetesRun
from prefect.executors import DaskExecutor
from prefect.storage import GitLab

@task
def inc(x):
    sleep(random.random() / 10)
    return x + 1

@task
def dec(x):
    sleep(random.random() / 10)
    return x - 1

@task
def add(x, y):
    sleep(random.random() / 10)
    return x + y

@task(name="sum")
def list_sum(arr):
    return sum(arr)


with Flow("dask-test") as flow:
    incs = inc.map(x=range(100))
    decs = dec.map(x=range(100))
    adds = add.map(x=incs, y=decs)
    total = list_sum(adds)

flow.storage = GitLab(<git kwargs>)
flow.run_config = KubernetesRun(image="prefecthq/prefect:latest-python3.8")
flow.executor = DaskExecutor("<tcp://dask-scheduler:8786>")

flow.register(project_name="test")

When running this flow with a

LocalDaskExecutor

, it works fine. In the Kubernetes cluster, however, the job is never marked as finished on the Prefect server. I can see the job execute to completion on the Dask scheduler UI, but the flow run goes indefinitely on Prefect. Any help would be greatly appreciated!

Hi there! Firstly, I'm sorry if this is a repeated question, but I have been banging my head against a wall for some time now. We have Prefect Server deployed to a machine on our company cluster using docker. I have a flow (example.py) that runs on the same machine happily by running Prefect locally. I can start an agent in the directory of the script (or define the path to my script with -p) and register the flow. Flows are defined in ~/.prefect/flows/ (not where I am running the script nor the agent) and when registering the flow, it appears in the ui just fine. However, when I try to run the flow from within the ui, I get:

Failed to load and execute Flow's environment: ValueError('No flows found in file.')

Any ideas what I am doing wrong?

Hello! I’m having a hard time with containerizing my prefect docker agent. I will post more information in the thread

sorry to keep posting here but im still having problems getting this to run on a VM. i have tried both inside a venv and outside. Googling doesnt seem to help. Prefect installs with no errors but running

prefect server start

keeps giving me this error. Is there somewere i can just get a copy of the docker-compose file? I’ve tried this on two VMs now

ERROR: 
        Can't find a suitable configuration file in this directory or any
        parent. Are you in the right directory?

        Supported filenames: docker-compose.yml, docker-compose.yaml

i had a look in the directory mentioned in the traceback for the docker-compose.yml and it is there, so I’m not sure why it isn’t being recognised

Traceback (most recent call last):
  File "******/venv/lib/python3.8/site-packages/prefect/cli/server.py", line 608, in start
    subprocess.check_call(
  File "/usr/lib/python3.8/subprocess.py", line 364, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['docker-compose', 'pull']' returned non-zero exit status 1.

Running Prefect Server on K8s and the apollo server pod keeps crashing with this error message:

PayloadTooLargeError: request entity too large
    at readStream (/apollo/node_modules/raw-body/index.js:155:17)
    at getRawBody (/apollo/node_modules/raw-body/index.js:108:12)
    at read (/apollo/node_modules/body-parser/lib/read.js:77:3)
    at jsonParser (/apollo/node_modules/body-parser/lib/types/json.js:135:5)
    at Layer.handle [as handle_request] (/apollo/node_modules/express/lib/router/layer.js:95:5)
    at trim_prefix (/apollo/node_modules/express/lib/router/index.js:317:13)
    at /apollo/node_modules/express/lib/router/index.js:284:7
    at Function.process_params (/apollo/node_modules/express/lib/router/index.js:335:12)
    at next (/apollo/node_modules/express/lib/router/index.js:275:10)
    at cors (/apollo/node_modules/cors/lib/index.js:188:7)

Any idea where to start for debugging that?

Hello all, any chance someone can offer advice or reference examples on spinning up a Kubernetes deployment on Google Cloud Platform as a preliminary step to deploying ephemeral DaskExecutor flows? My org has a limited budget and we’re trying to minimize cloud costs by scaling our infrastructure to zero when we’re not running large web scraping jobs once or twice a week. I’ve looked at Dask Cloud Provider, which has GCP support and could work. But we also wanted to explore a k8s solution before settling on a final strategy. I realize this question is not directly related to Prefect, but hoping someone might have experience with this kind of approach (perhaps using something like Terraform?). Any advice is greatly appreciated!

Hi everyone, I'm creating a basic docker-compose setup of prefect + dask. I'm able to run flows with LocalExecutor in Prefect, and I'm able to run tasks in Dask directly, but I'm not able to run Prefect flows using DaskExecutor. My flow runs simply stay in a pending state and never execute. I don't see any errors in the logs or dashboard. What are some things I can try to debug for issues? I can share my docker-compose file if that helps. The only possibly unusual thing I can see is that I'm running a the agent as 'local' although it is running within a container, but I think that should be fine?

When I inspect the agent logs (run with "prefect agent local start" inside a container), it looks like it is only picking up flow-runs with LocalExecutor. I see there is also LocalDaskExecutor but I don't want the agent to run its own local dask cluster, I want it to send work to the scheduler, which is in a separate container.