hey prefect we had some strange behaviour with our on prem prefect v1 stack, one hosted in aws and another in azure where flows will not run and we also had an unusual amount of Postgres activity, these are completely isolated from each other too would any issues with prefect cloud cause potential problems with the community v1 on prem solution?

Hi! I was thinking of using Prefect but I'm not sure if it works well for my task. I am running large pools of GPU workers nodes which are orchestrated via some separate set of logic, and latency matters a lot. My thought was to set up Kafka as a messaging queue that all these workers listen too since it also offers the ability to do kSQL joins which are beneficial for me to say match up frames and frame metadata that might've been processed by different models upstream. I was also concerned about somehow having a warm pool of workers so the models don't need to cold start every time. I wouldn't even think about a workflow orchestrator but I want to have the benefits of task observability, logging, etc which it feels I wouldn't get if I used Kafka by itself.

All, I am running a flow using prefect UI and hitting a

ModleNotFound

error. I register the code as follows

prefect register --project CHX -p sparsify.py -n "sparsify" --label chx-2022-2.2

The folder where

sparsify.py

sits has the missing module’s python code; masks.py. And I import the module and relevant functions as follows

from masks import MaskClient, combine_masks

Somehow when I run sparsify via UI it can’t find it. The code just runs fine in an

ipython

session

Hello, the doc mentions that the .prefectignore file supports the git ignore patterns. However when testing I see that this does not support wildcard like "*" exclude everything. Am I the first to face this ?

Hello, Prefect community! I have a question. I’m considering using ECSTasks for my execution layer setup, but wanted to verify something first. From www.prefect.io/guide...#NewDockerdeploymentexperience

If your deployment defines

-ib docker-container/prod

and doesn’t specify any remote storage block, Prefect will assume that the flow code is baked into the image.

Does this hold true for

-ib ecs-task/prod

as well?

Hi Everyone! I am looking for some advice on Prefect-DBT Core integration. My company uses Prefect version 1. I initially started with cloning DBT repo into Prefect method(referred online resource) which worked fine locally but not inside the Prefect cloud as our internal org repos have strict security policies that don't allow them to be cloned inside 3rd party products. Hence, I decided to try out the git submodule method where the DBT repo resides as a submodule inside Prefect repo and I use DBTShellTask to execute the DAG commands(like below). The flow works fine in my local but in Prefect, it fails due to statement helper_script=f"cd { DBT_DIR}" as it is unable to access the right folder. It evaluates helper_script as (cd: /Users/my-username/Development/prefect-repo/containers/dbt-prefect-container/dbt-repo)

dbt = DbtShellTask(
        return_all=True,
        profile_name="dbt_repo",
        environment="dev",
        # profiles_dir=".",
        overwrite_profiles=True,
        helper_script=f"cd {DBT_DIR}",
        log_stdout=True,
        log_stderr=True,
        dbt_kwargs={ # removed for readability}
        )(command="dbt docs generate")

The folder structure is like: prefect-repo |- containers |- dbt-prefect-container |-dbt-repo(submodule) |- flows |-flow.py (from where I want to access the location DBT_DIR) My questions are: 1. Is it okay to use git submodule method inside Prefect? 2. How to pass correct repo location so that Prefect cloud doesn't start the root at local dir: /Users/my-username

How to set the time threshold for a flow run to be considered "Late"? (e.g. whether to wait 30 seconds or 30 minutes before marking a flow run as late) Related previous posts (none of those was ever answered): • https://prefect-community.slack.com/archives/CL09KU1K7/p1670260486146599 • https://prefect-community.slack.com/archives/CL09KU1K7/p1670246788470149 • https://prefect-community.slack.com/archives/CL09KU1K7/p1668495473066159

hello! is it possible to build/use custom infrastructure types (beyond what’s available in the

prefect.infrastructure

module)? I assume no since this needs to be present in the agent’s env but double checking

Hi, In Prefect 1 we often see flows stuck in Running state because the underlying k8s pod might have been terminated. Prefect is for some reason not able to detect this and the flow stays in a running state. In one case the logs show

Flow run RUNNING: terminal tasks are incomplete.

as the last log message, then flow stays in running state forever. Is this a known issue?

Hi Team, How do i set infra_override in the deplyments yaml to override my ECS task memory for a particular deployment. Can someone please help me with this?

hey guys, I’m trying to implement an exponential backoff retry in a @task where I use native python http Retry class, however in the execution of the task I don’t see the retry being invoked in the Flow? can someone enlighten me on this one?

Hi ! i was trying to write a flow which runs a deployment (wrapped under a task) parallely for a set of input ( in array) . and i want to perform certain actions only after all these runs have completed.

from prefect import task, flow
from prefect.deployments import run_deployment


@task()
def task_A(event):
    output = run_deployment(
        name='some deployment',
        parameters={'event': event}
    )
    if output.state_name != 'Completed':
        raise Exception("flow run not completed")
    return output


@flow()
def flow_B():
    a = [1, 2, 3, 4, 5, 6, 7]
    # method 1
    for i in a:
        task_A.submit(i)

    # method 2
    output = task_A.map(a)
    for tasks in output:
        tasks.wait()

    #do something when all the parallel executions have completed for task A

this is what my flow looks like but in method 1 it runs next iteration of task_A only when previous iteration is complete in method2 it doesnt wait for the tasks to complete

Hi, with Prefect 2.0, how to use command line to run a flow with parameters (prefect deployment run id ......etc)

I'm looking to try to use a 3rd party secret store (we use Hashicorp Vault, but AWS Secrets Manager, etc would be similar). In our blocks, we want to store a pointer to the secret (path, location, etc), but not the actual value. And then retrieve the value at runtime. Has this been done before?

Hi guys! I'm running a flow using the Orion dask executor on Kubernetes but i get an error on very large datasets:

distributed.scheduler.KilledWorker: Attempted to run task prometheus_to_gcs-02d33e7b-92a6-4fe8-8258-ba9efaa3d609 on 3 different workers, but all those workers died while running it.

What could be the problem here? Other smaller datasets are running fine. I am running it with adaptive workers (min 4, max 10). I am not sure though if I used correctly to dask configuration for what i am trying to do. Can you please have a look at my code in the thread? I am able to run the same flow in Prefect 1.4. Thanks!

Hey Folks, how can I check the health of a Perfect Agent in 2.0? It is running inside a docker container and I’m trying to figure out a reliable health check locally instead of pinging the Orion server.

Does prefect 2.0 have a way to tell a task to wait for manual intervention? For workflows with humans in the loop.

deploying Azure Container Instances - I keep getting a timeout waiting on container creation. Has anyone else experienced this?

After installed Prefect Orion 2.7, I am getting the following exception and I couldn't see any of my existing flow-runs in Orion UI. TypeError: unsupported operand type(s) for -: 'DateTime' and 'NoneType'

I have a flow that is occasionally failing with the following error: State message: Given states: total=2, not_final=2, 'RUNNING'=1, 'PENDING'=1. Both of the tasks in the flow finish as completed as seen in the photo. The flow's code looks like this:

@task
def prod_9e_etl_asx_anns():
    logger = get_run_logger()
    res = invoke_lambda("prod_9e_etl_asx_anns", {}, True)
    <http://logger.info|logger.info>(res)
    return ""


@task
def prod_9e_etl_asx_ann_sentiment():
    logger = get_run_logger()
    res = invoke_lambda("prod_9e_etl_asx_ann_sentiment", {}, True)
    <http://logger.info|logger.info>(res)
    return ""


@flow(task_runner=SequentialTaskRunner(), retries=2)
def perfom_asx_anns_etl():
    x = prod_9e_etl_asx_anns.submit(return_state=True)
    y = prod_9e_etl_asx_ann_sentiment.submit(wait_for=[x])

I need to make sure the tasks happen in sequential order. Any idea how I prevent this error?

I’m trying to run

prefect agent -q tenzinga_django_prod

on my server … I’m using a

systemd

to run. It errors and lets me know that Orion (the server not agent) needs SQLite >= 3.24.0 but Amazon Linux 2 only has 3.7.17 as it’s latest version.

This message was deleted.

@Anna Geller Is there a plan to integrate celery for task distribution in roadmap?

@community, anyone used celery with Prefect for task distribution?

Dose anyone know the status of the prefect 1.0 API? I have an internal error code 500 while requesting prefect 1.0 api.

Are there any best practices when it comes to setting up a dev and production env in prefect? I was thinking to put it in two different workspaces, so it would also be easier to take advantage of blocks, but I'm not sure if i could run two agents on two different workspaces on the same pc. (Would have to login to two different workspaces simultanously?) Currently i've setup two different docker blocks that injects the secrets i need in different environment into the env vars in the container, but this would lead to exposed secrets if i was to inject secrets into the docker like this. One way would be to use different Azure vaults and just inject these into the docker, but I believe the prettier solution would be to use prefect blocks. Anyone who did something similar and can share ideas or experiences? TL:DR - What is the best way to deal with secerts/connection string and other variables that span across environments? Ex.. The Server1 Var in my Dev environment == DevServer Server1 Var in Prod var == ProdServer

Hi what is minimum hardware requirements (CPU/RAM) for Prefect production installation ? with all components (SQL/UI/Engine/Agent etc.)

Hi all, Is there a way of retrieving a flow's scheduled start time from context in Prefect2 ? As we want to use the scheduled time (not the actual run time) as input for some of its tasks. It was easy in Prefect 1 but can't work out how to do it in Prefect2 now that we've migrated. Thank you !

Thanks Anna! Sorry about that, i'll keep that in mind next time :)