Hello Prefect !! I am Ravi, and I am working as a NLP Engineer at Enterpret. Currently looking to scale the data processing pipelines and have a way to monitor them. I am looking into prefect for the workflow automation. Looking forward to learn and contribute to the community 😃
Hey everyone! I'm Richard, a Data Science Evangelist intern over at Coiled.io. Currently exploring all the ways our product integrates with the wider PyData ecosystem and naturally arrived at Prefect. Looking forward to learning more about the product and community here!
Hi, I'm Mathias, co-founder and CTO of SenX, maker of Warp 10 (https://warp10.io/), a Geo Time Series platform. We have used various tools for task automation along the years, Luigi then Airflow then Prefect.
Hi all 👋, I'm Cory from Austin FC ⚽🤓, new to slack but I've been using prefect since April or so... hopefully i can help folks as much as asking annoying questions... I'm using prefect with ❄️ snowflake to load data about soccer games around the world 🌍 daily and run a lot of our internal processes that drive a web app for coaches, scouts, and managers!
Hello - I'm the Analytics & ML lead at Netdata Cloud and looking at prefect cloud as a potential alternative to some flavor of Airflow (self managed, composer or ideally Astronomer).I'm having a little trouble understanding what exactly agents are or what i would need - kinda feels like agents are almost like airflow workers but not quite. If i wanted to use prefect cloud to manage the orchestration of BigQuery jobs and GCP cloud function based pipelines. It seems like maybe i would still need to bring my own agents with me - is that correct? If so do you have any typical examples of how people might do this?From what i have read it looks like maybe i'd ideally have my own dask or kubernetes cluster to have lots of agents available to prefect cloud? Just wondering how to think of scaling this - how would i know how many agents i would need?Or i may be getting things wrong in my understanding perhaps?p.s. apologies if this not a good place for this question. Oh and hello again 🙂
Hi, I'm trying to figure out if I can use Prefect for an ETL-type scenario. Basically, we'll be receiving large text files via SFTP/HTTPS and we need to process these files eventually into a data warehouse. The Prefect documentation is a bit confusing to me as it mostly focuses on Cloud. I am primarily interested in an on-premise deployment. Would appreciate if anyone can point me to resources that could help me answer these questions:1. Is Prefect Core = Server + UI?
2. Is there a deployment guide for on-premise that doesn't require Docker/Compose?
1 year ago
Hi all,We are currently experimenting with prefect as to replace our own orchestration tool used for scaling complex compute tasks. 👋
I’m here to migrate from airflow to prefect because of the latency.I have some questions here.I have 2 layers for the process.1. data processing
I have many data processed every minute and also have many models processed every minute.A model have dependencies to some data.
data_A - model_A
data_B -data_A - model_B
data_C -I tried to make each data processing into a flow -> N flows
and also each modeling process into a flow -> M flows
Then I had a parent flow setting dependencies between data flows and model flows.
However this had long latency (I suspect poll_interval)So I made a huge flow which contains all data processing and modeling process.
this made the flow much faster but hard to monitor with the UI.Do you think there are some way to achieve this goal?