Hello all, I have a question about prefect. Currently I have an abstract factory which instantiates a bunch of products and each one of those products I want them to ideally contain a flow, but reading about how prefect works it doesn't really allow for OOP based flows? Any advice on how I can add it do our existing structure?

Hi i am receiving this error when i am running prefect on my mac, any ideas?

Traceback (most recent call last):
 File "/Library/Frameworks/Python.framework/Versions/3.10/bin/prefect", line 5, in <module>
  from prefect.cli import app
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/prefect/_init_.py", line 45, in <module>
  from prefect.engine import pause_flow_run, resume_flow_run
 File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/prefect/engine.py", line 98, in <module>
  from anyio import start_blocking_portal
ImportError: cannot import name 'start_blocking_portal' from 'anyio' (/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/anyio/_init_.py)

Newby question. I am installing version 2.11.5 in my python venv. I get the following error however.

ImportError: cannot import name 'start_blocking_portal' from 'anyio' (/home/gtracy/prefect-env/lib/python3.10/site-packages/anyio/__init__.py). I'm using Python 3.10.12.

Thoughts?

Hello, I'm creating my first flow run on local for the first time, and i want to set it up with how (i think) my prod set up will interact with prefect. I have a work pool created and a local agent listening to it, and I deployed my Flow to prefect cloud like so:

if __name__ == "__main__":
    
    deployment: Deployment = Deployment.build_from_flow(
        flow=test_flow,
        name="example",
        version="1",
        tags=["demo"],
        work_pool_name='local_agent_work_pool'
    )

    deployment.apply()

I confirmed that the deployment worked in the Prefect Cloud UI. In my flask endpoint, I am trying to submit the flow run to Prefect cloud like so, but it feels wrong:

@flask_route.route("/prefect", methods=['POST'])
def start_prefect_flow():
    # prefect_client = get_client(
    #     # api_key='redacted'
    # )
    prefect_client = PrefectClient(
        api='redacted',
        api_key='redacted'
    )

    res = prefect_client.create_flow_run(test_flow)
    print(res)

First, I don't think I should be creating the client this way, but unsure how else I should be passing my api/api key information to it. This is also returning a coroutine object instead of what I thought would be information about my flow run, but maybe that makes sense since I don't see a flow run being kicked off. Some assistance here would be super helpful!

I just installed Prefect 2.12.0. I can get the dashboard to come up using 127.0.0.1:4200, however when I try using another machine to access the dashboard I cannot connect. I am running Prefect on my PopOs ubuntu server. I did a lsof -i :4200 and got no response. So I believe that Prefect is not setting up port 4200 to be listening outside. Thoughts? Thanks. Still a newbie with Prefect.

Hi all - I'm trying to set up a prefect2 agent following this tutorial https://discourse.prefect.io/t/how-to-deploy-a-prefect-2-0-agent-to-an-ec2-instance-as-your-execution-layer/551 My flows use prefect-aws for infrastructure so I'm getting the following error when I launch my flow - I think because prefect-aws is not installed on my instance?

KeyError: "No class found for dispatch key 'ecs-task' in registry for type 'Block'."

If I run

pip3 install prefect-aws

on my instance I get this dependency conflict:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
aiobotocore 2.5.4 requires botocore<1.31.18,>=1.31.17, but you have botocore 1.31.40 which is incompatible.

And when I launch my flow I get the same KeyError along with:

Warning!  Failed to load collection 'prefect_aws': AttributeError: module 'lib' has no attribute 'OpenSSL_add_all_algorithms'
17:32:19.694 | ERROR   | prefect.agent - Failed to get infrastructure for flow run '96a4ce05-7431-4062-bb2b-8c039d5d77e0'.

Are there versions of prefect2 / prefect-aws I can use to get this to work? Am I missing something obvious / should I rethink my execution layer?

Is there a simple way to write a schedule that kicks a flow run after a given time interval that the previous run has finished (not started)?

What is the best way to deal with env vars for kube workers? Should we handle that in our Dockerfile, or is there a programmatic way of sending env vars to the execution pods?

Ping on the above! I have a local worker running but the flow requires a pull from s3 (and other env related things), so wondering how I can pass env vars to workers in local and prod

Hi is there a prometheus exporter for prefect 2? i can see in prometheus exporter that it requires a graphql URL but the Prefect 2 doesn't have it. prometheus exporter chart https://github.com/technicityworks/helm-charts/tree/main/charts/prometheus-prefect-exporter (edited)

I need help knowing how I can limit the number of slack execution records in the prefect database, so that the database does not fill up. I want to see if I can limit some flows to that. or save more than 5 days. Thanks for the help you can give me

Hi Team, I have an issue with job scheduling. When I schedule around 50 jobs in parallel, 10 jobs are going to a crashed state. Previously re-running the crash jobs would fix but though there is no load a few of the crash jobs are continuously going to crash state itself. Why is this frequent crashing happening and how can I fix this.

Hi all, I need to create more than 3 automations in one workspace but prefect ui says only 3 automations are allowed in one workspace. Is there a way to do this in one workspace or do I need to create another workspace?

Hi all, new to prefect. I would like to load and apply profile / settings from a profile.toml and a logging.yml during runtime that is part of my repo, how would I best go about it / what is best practice? I've not found a viable solution via the docs or community threads. Any help is much appreciated.

message has been deleted

Hi there! Apologies in advance for how long this is. I’m working on a project and I am somewhat confused when it comes to Deployments, Work Pools, and Workers. I have my Flows successfully created and have tested them from my local. Everything’s working great, easily connected into Prefect Cloud. I’ve created a Block that has my GitHub repo information, and have added that as the Storage for my Deployment. My Deployment is currently sitting on my local and hasn’t been pushed up to Prefect Cloud, mainly because I’ve been trying to figure out how to run this on a GCP VM. I was thinking initially about using a Process Worker and just having the daemon keep the process running, but I’m having issues getting the worker registered with systemctl (I read the article, still no dice) and I also have other stuff running on the server, so I’d prefer to keep it separate. So then I thought I could install a Prefect Docker container and run the worker from there, but I can’t get into the container for some reason. So now I’m leaning towards a Python container with Prefect installed on it which runs the Worker. Seems like this might work, but I’m just starting to investigate that path. What I don’t understand is are the other Docker alternatives and how to use them. For instance, I’ve read this about 15 times, but I’m still unsure about what I need to do. So here are some questions - what does prefect.yaml do and how is it different from my deployment? Do I need both prefect.yaml and my deployment file? If I want to build ephemeral Docker containers, how do I do that? Should I use a Docker Work Pool? If I do that, will it automatically build the container, pull the repository, and install the dependencies before running the Flows? Or do I need to build a container that has my Flows in it which then gets pulled in (I’m trying to avoid this option)? To be honest, everything up to this point has bee very clear cut and straightforward, but I just can’t seem to get my head wrapped around this concept for some reason. Any help would be appreciated!

Hi! I’m establishing pipeline using prefect for my team. Have a question about task function argument type limit, since I did not find any reference in doc. Detail shown in my git discussion: https://github.com/PrefectHQ/prefect/discussions/10734

Hi Team, Is there any way I can purge the prefect database. I want to schedule this databse purge every month. Can anyone please help me on this

My organization today is using a home-made data orchestration tool built on top of RQ in Python. I'm evaluating Prefect as an alternative, but from what I can tell workers are blocking. Is my understanding correct? E.g. a flow should run 10 tasks, some of which wait for the results of others. If I only have a single worker, is it impossible for me to be running 2 instances of this flow concurrently?

hey friends! anyone know how to delete artifacts with code? I only see ways to create

potentially foolish question: is there any way i can run a prefect task fn outside of a flow /t ask run?

Hi team! How can we limit task concurrency when using .map() with 1000+ items? Especially when running local flow runs (with remote server), it's currenly not a great experience. Prefect starts all of these tasks at the same time, which clutters both logs and web ui. I've read about the new global concurrency limits and the concurrency context manager, but 1) afaik it only limits concurrency within the tasks, not at the .map level (we see all the tasks started, waiting for their slot, pretty bad in the web ui), 2) the concurrency is global and not per-flow-run, which doesn't make sense when running flows locally (and arguably remotely too for a lot of cases). How do you deal with this in your projects?

hey, does anyone know of a way to set the path for where the

prefect.yaml

file is? i have a scenario where this file will get recreated each time a container gets spun up - but i also want to run it with volumes. and i dont want each container replacing the other's file

Hi folks, forgive the stupid question but what linux user is the prefect agent running with? I am unable to find any info of this. I need to give permissions to the existing user and don't want to use root ofc. Thanks!

Hi team, I'm trying out Prefect. Can someone explain like I'm 5 how do I go from flow/task to deployment and why the task actually run on my computer after I ran the serve? I figured I should make a project like other tools and then submit the package to Prefect to run on Prefect server, is it wrong? It seem to me I have to open a CMD and let it hang around every time I deploy a new project(?).

Hello Team, I'm very to new prefect and would want to deploy our production ml pipeline in AWS ECS. I have been going through a lot of docs, blogs and videos, but seems little overwhelming to me. I would like to take the guidance from experts who would have already setup production pipelines in AWS , to share the advise on the areas that I could focus and plan the deployment. Help would be much appreciated !! I have run the flows and tasks locally and test the prefect server, but now i want deploy it in AWS and Im finding it little difficult. I have tried for follow few blogs but getting some unexpected errors while starting the prefect server. https://towardsdatascience.com/deploying-prefect-server-with-aws-ecs-fargate-and-docker-storage-36f633226c5f. Thanks again 🙂

Hi I am trying to run the Prefect pipelines in k8s. I am using the yaml config for build, push and pull. I got the worker to run, however, when I try to create a pod in k8s to execute the pipeline I am getting "NO_CERTIFICATE_OR_CRL_FOUND". Could you please inform • How can I disable this check for dev env ? • How can I provide the cert in higher env via prefect yaml ?

Failed to submit flow run '***' to infrastructure.
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/urllib3/util/ssl_.py", line 402, in ssl_wrap_socket
    context.load_verify_locations(ca_certs, ca_cert_dir, ca_cert_data)
ssl.SSLError: [X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4154)

Hello..Everyone! Im new here and looking for a Product guide or Power Point that talks about Prefect. Can you share if you have one