https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
introductions
  • r

    Ramzi A

    08/13/2021, 2:33 PM
    👋 Hello everyone! I am a Data Engineer at Flex a fintech company in New York. Lurking here to learn more about Prefect!
    🙌 1
    👋 12
    k
    d
    +3
    • 6
    • 5
  • t

    Tim Chklovski

    08/15/2021, 12:22 PM
    👋 Hi! I’m running a nascent fund automatically trading market-moving news. Want to learn about using Prefect+more to run all our ETL jobs (sync price data, news feeds, retrain ML models). A bit more at predmachine.com. Thanks for a making this community great!
    👋 13
    j
    d
    +3
    • 6
    • 5
  • w

    Wilson Bilkovich

    08/17/2021, 3:43 PM
    Cheers; I work at a fintech company, and we’re using Prefect for our precompute/research platform. Youtube videos have been useful, thanks for that!
    👋 15
    k
    n
    +2
    • 5
    • 4
  • y

    Yves

    08/17/2021, 6:05 PM
    Bonjour! Just checking out Prefect and couldn’t resist the temptation to join another Slack workspace. 😉 Looks like a nice community!
    👋 11
    n
    k
    j
    • 4
    • 11
  • e

    Eddie Atkinson

    08/18/2021, 5:27 AM
    G'day to all. I work at a green energy startup looking at using Prefect to schedule ETL jobs for IoT devices. Been easy to get started with Prefect, I just have some questions about best practices for deployment 🚀
    👋 13
    n
    j
    +3
    • 6
    • 5
  • g

    Guillaume Legoy

    08/19/2021, 10:15 AM
    Hello folks, Guillaume from Bandai Namco Mobile here. Trying out Prefect as an alternative solution to Airflow for our data pipelines needs, thanks for creating this community!
    👋 13
    c
    j
    +3
    • 6
    • 5
  • g

    Gaylord Cherencey

    08/23/2021, 2:13 AM
    Hi all, Gaylord working for Xero in New Zealand. Considering Prefect as an orchestration tool and I wanted to know if this Slack is also a good place to ask questions around Prefect Cloud?
    👋 11
    e
    a
    +3
    • 6
    • 7
  • m

    Matt Alhonte

    08/25/2021, 11:02 PM
    Matt Alhonte again, reintroducing myself a 3rd (4th?) time now that I'm on a new Prefect job as a Machine Learning Engineer with SymphonyRM! 🙂
    👋 12
    n
    n
    +4
    • 7
    • 6
  • h

    Henning Holgersen

    08/26/2021, 6:27 PM
    Hi, I’m Henning. I’m a free-range data clown based in Norway. Trying out Prefect in order to collect scattered lambda functions (across two clouds) into something more sane. By the way, what is the group name for lambdas? It’s a murder of crows, a gaggle og geese, a skulk of foxes, a … of lambdas?
    👋 14
    k
    g
    +3
    • 6
    • 9
  • d

    Donny Flynn

    08/31/2021, 5:47 PM
    Hi, I'm Donny working and I am a Customer Data Architect at Census. I have been playing around with Prefect and am a former head of data (engineering, science, analysis) at two early-stage companies. I hope to contribute to the community around this awesome product!
    🎯 2
    👋 12
    k
    m
    +4
    • 7
    • 6
  • b

    Ben Castleton

    08/31/2021, 11:47 PM
    👋 - Hi everyone. Ben here from the Great Expectations team. I’ve been hearing so many good things about the Prefect community for so long that I finally figured I should join to check it out! Awesome work. I’m way impressed with what ya’all have accomplished so far. Everyone I talk to that’s used it loves Prefect. Doing something right here! 🎉
    ❤️ 9
    👋 17
    n
    m
    +5
    • 8
    • 7
  • a

    Abhishek

    09/02/2021, 7:07 AM
    Hello folks 👋 , Abhishek here, currently based in India. I am a Python developer and currently working on few data engineering tools and Infra. Apart from my work I also organize community events across India which includes local Python meetups and PyCon India conference. Glad to be a part of this vibrant community! Looking forward to contribute 🙂 Prefect is perfect 🙂
    👋 10
    j
    c
    +4
    • 7
    • 6
  • j

    John Shearer

    09/02/2021, 8:46 AM
    Hello Prefect World! Great to be here. I'm John, currently working on some data pipelines for Altium.com and Octopart.com (PCB design software and electronic components search, respectively). After spending a couple of months with Airflow I tried out Prefect, and rebuilt 2 months of work in a couple of days! Ecstatic 🙂 ❤️ super impressed, and love the community so far.
    👋 11
    j
    c
    +3
    • 6
    • 6
  • b

    Bryan Cusatis

    09/02/2021, 1:50 PM
    Hello Prefect! I'm Bryan, VP of Architecture at an Ed Tech company specializing in the data/backend side of things. Evaluating Dagster and Prefect for our ETL needs. Happy to be here!
    👋 14
    j
    z
    +3
    • 6
    • 6
  • b

    Blake Brienz

    09/02/2021, 2:23 PM
    Hello, I'm here exploring my options other than Airflow. 😄
    👋 15
    k
    j
    +2
    • 5
    • 4
  • h

    Hari Krishna Sunkari

    09/03/2021, 4:54 AM
    Hello Prefect! , Im Hari, exploring argo, prefect & temporal. we have used airflow before, but started evaluating these to look for something beyond just scheduling.
    👋 10
    👍 2
    k
    n
    +2
    • 5
    • 4
  • j

    Janne Keskitalo

    09/05/2021, 7:53 AM
    Hi. I'm Janne, Head of Data Engineering at CollectiveCrunch. I'm looking into Prefect to run our petabyte -scale geospatial data processing pipelines.
    👋 10
    j
    j
    k
    • 4
    • 3
  • t

    Trip Humphrey

    09/05/2021, 9:45 PM
    Howdy everyone! I'm Trip, and I work as Healthcare Data Consultant in Virginia. I have about 15 years experience in Healthcare, IT, and Data. Currently I am evaluating a number of solutions that pertain to data workload orchestration. I am curious about prefect, but I will admit I am having a bit of trouble parsing the seemingly nuance differences between Flow Storage and the concept of version control of the Flow .py files themselves. Do we use a traditional git repo with cicd workflows providing change control of the actual code? I know I can store the flows in a number of different ways, but I just can't seem to get it to click as to how a team might collaboratively manage updates to those flows over time.
    👋 11
    m
    • 2
    • 7
  • r

    Raviraja Ganta

    09/07/2021, 6:17 AM
    Hello Prefect !! I am Ravi, and I am working as a NLP Engineer at Enterpret. Currently looking to scale the data processing pipelines and have a way to monitor them. I am looking into prefect for the workflow automation. Looking forward to learn and contribute to the community 😃
    👋 13
    c
    d
    +3
    • 6
    • 7
  • r

    Richard Pelgrim

    09/08/2021, 8:58 AM
    Hey everyone! I'm Richard, a Data Science Evangelist intern over at Coiled.io. Currently exploring all the ways our product integrates with the wider PyData ecosystem and naturally arrived at Prefect. Looking forward to learning more about the product and community here!
    👋 12
    🐍 3
    c
    j
    +4
    • 7
    • 7
  • m

    Mathias Herberts

    09/08/2021, 9:38 AM
    Hi, I'm Mathias, co-founder and CTO of SenX, maker of Warp 10 (https://warp10.io/), a Geo Time Series platform. We have used various tools for task automation along the years, Luigi then Airflow then Prefect.
    ⏳ 1
    👋 11
    n
    c
    +5
    • 8
    • 7
  • c

    Cory Jez

    09/08/2021, 1:09 PM
    Hi all 👋, I'm Cory from Austin FC ⚽🤓, new to slack but I've been using prefect since April or so... hopefully i can help folks as much as asking annoying questions... I'm using prefect with ❄️ snowflake to load data about soccer games around the world 🌍 daily and run a lot of our internal processes that drive a web app for coaches, scouts, and managers!
    👋 12
    ⚽ 7
    🔥 8
    j
    c
    +4
    • 7
    • 7
  • w

    Wieger Opmeer

    09/10/2021, 12:21 PM
    Hi all, I'm currently trying to figure out if we should stick with Airflow or byte the bullet and start using something else.
    👋 12
    m
    j
    +2
    • 5
    • 4
  • p

    Prateek Gupta

    09/11/2021, 6:24 PM
    Hi Everyone. I have been asked to deploy ML models using Prefect.
    👋 14
    z
    j
    +3
    • 6
    • 5
  • a

    Andrew Maguire

    09/13/2021, 2:33 PM
    Hello - I'm the Analytics & ML lead at Netdata Cloud and looking at prefect cloud as a potential alternative to some flavor of Airflow (self managed, composer or ideally Astronomer). I'm having a little trouble understanding what exactly agents are or what i would need - kinda feels like agents are almost like airflow workers but not quite. If i wanted to use prefect cloud to manage the orchestration of BigQuery jobs and GCP cloud function based pipelines. It seems like maybe i would still need to bring my own agents with me - is that correct? If so do you have any typical examples of how people might do this? From what i have read it looks like maybe i'd ideally have my own dask or kubernetes cluster to have lots of agents available to prefect cloud? Just wondering how to think of scaling this - how would i know how many agents i would need? Or i may be getting things wrong in my understanding perhaps? p.s. apologies if this not a good place for this question. Oh and hello again 🙂
    👋 10
    k
    k
    +2
    • 5
    • 8
  • j

    Joseph Harrison

    09/14/2021, 12:11 AM
    Hi, I'm trying to figure out if I can use Prefect for an ETL-type scenario. Basically, we'll be receiving large text files via SFTP/HTTPS and we need to process these files eventually into a data warehouse. The Prefect documentation is a bit confusing to me as it mostly focuses on Cloud. I am primarily interested in an on-premise deployment. Would appreciate if anyone can point me to resources that could help me answer these questions: 1. Is Prefect Core = Server + UI? 2. Is there a deployment guide for on-premise that doesn't require Docker/Compose? Thanks!
    k
    d
    • 3
    • 8
  • l

    Lucas Beck

    09/14/2021, 8:32 AM
    Hi all, We are currently experimenting with prefect as to replace our own orchestration tool used for scaling complex compute tasks. 👋
    👋 11
    d
    j
    t
    • 4
    • 3
  • b

    Baum

    09/14/2021, 10:28 AM
    Hi all, I’m here to migrate from airflow to prefect because of the latency. I have some questions here. I have 2 layers for the process. 1. data processing 2. modeling I have many data processed every minute and also have many models processed every minute. A model have dependencies to some data. i.e) data_A - model_A data_B - data_A - model_B data_C - I tried to make each data processing into a flow -> N flows and also each modeling process into a flow -> M flows Then I had a parent flow setting dependencies between data flows and model flows. However this had long latency (I suspect poll_interval) So I made a huge flow which contains all data processing and modeling process. this made the flow much faster but hard to monitor with the UI. Do you think there are some way to achieve this goal?
    👋 6
  • b

    Bob De Schutter

    09/15/2021, 10:08 AM
    Hi there, my name is Bob and I'm currently looking into prefect as orchestration engine for a new data platform I'm setting up at a client. I have a lot of experience with Airflow, so I'm curious to see how Prefect does the job 😉
    👋 11
    j
    c
    +4
    • 7
    • 6
  • d

    Dominic Pham

    09/15/2021, 8:11 PM
    Hi folks, my name is Dominic and I'm a budding data engineer trying to implement a data pipeline ecosystem at my company for webscraped and collected data. I'm an experienced Alteryx user trying test out more open source python based solutions, and the many pain points I've experienced even trying to start Airflow has led me to try out Prefect instead.
    👋 12
    z
    m
    +2
    • 5
    • 4
Powered by Linen
Title
d

Dominic Pham

09/15/2021, 8:11 PM
Hi folks, my name is Dominic and I'm a budding data engineer trying to implement a data pipeline ecosystem at my company for webscraped and collected data. I'm an experienced Alteryx user trying test out more open source python based solutions, and the many pain points I've experienced even trying to start Airflow has led me to try out Prefect instead.
👋 12
z

Zach Angell

09/15/2021, 8:21 PM
Welcome to Prefect @Dominic Pham! 👋
m

Michael Adkins

09/15/2021, 9:36 PM
Welcome !
j

justabill

09/15/2021, 9:56 PM
Hey @Dominic Pham, welcome!
t

Tyler Wanner

09/20/2021, 6:31 PM
Hi Dominic! Welcome to the community. Feel free to ask away, there are a lot of folks in this Slack who are committed to finding the best ways to run Prefect, and to making Prefect the best way to run data pipelines!
View count: 4