https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
show-us-what-you-got
  • m

    Marcus Hughes

    08/01/2022, 5:05 PM
    Code Contest What if we have many (and by many I mean MANY) subflows? In fact, let's just recursively create sub-flows while begging for a Prefect keyboard and/or a Marvin Rubber Duck 😆. See the first comment for code. As one might expect, Prefect runs into trouble when
    subflow_count
    (the number of subflows a flow generates aka breadth of the recursion tree) and
    levels
    (how many chained subflows there are aka the depth of the recursion tree) are large, 25 and 10 here respectively. That's 10^25 - 1 flows, truly absurd with long chains of subflows. When it fails, I get
    OSError: [Errno 24] Too many open files
    . I noticed that the crashed flows don't get marked as crashed but stay with the
    running
    status. I'm not sure there's an easy way for you to resolve that though. On the positive side, my computer didn't crash! At smaller values, e.g.
    subflow_count=8
    and
    levels=3
    , it runs just fine. My team and I are excited to use Prefect in the upcoming NASA PUNCH mission to prepare scientific data of the Sun. It's really helped make our pipeline more elegant.
    🪐 13
    🌞 12
    🚀 22
    🙌 1
    🧑‍🚀 11
    j
    c
    • 3
    • 6
  • j

    John Kang

    08/01/2022, 7:45 PM
    Code Contest I'm not showing the entire body of code, but have a long script that pulls data from databases, cleans it up, and pushes it to an online database to power some of our analytical applications. The only thing I'm waiting on is resolution on running a Prefect agent on Windows as occasionally the Prefect agent will just crash. After that is resolved I'm going to deploy this to a virtual machine. If it is not resolved within a week or two, I might just have to figure out how to run this on a RHEL virtual machine within our firewall..
    :cockroach: 4
    :blob-attention-gif: 2
    🚀 1
    🙌 7
    :party-parrot: 2
    :hattip: 1
    👻 1
    :marvin: 1
    c
    • 2
    • 2
  • s

    Stéphan Taljaard

    08/02/2022, 7:48 PM
    Code Contest I created a flow combining two things I have a strange fondness of: 1. Prefect flow run names 2. Craiyon ("DALL·E mini") images The flow takes the 5 latest flow run names and uses them as prompts for Craiyon. Though not really identifying any Prefect edge cases, it's an example of using: • Using native Python inside a flow • Using Prefect's logger • Mapping • .submit() • Interacting with PrefectFutures • Interacting with the API • Caching • Task concurrency limits --- Code here: https://github.com/taljaards/Prefect-2.0-code-contest
    :thank-you: 3
    :prefect: 3
    👏 7
    🔥 5
    🚀 15
    🎨 11
    :blob-attention-gif: 8
    🙌 5
    :marvin-duck: 5
    a
    a
    +4
    • 7
    • 18
  • c

    Chris L.

    08/03/2022, 10:28 AM
    Code Contest Just two async subflows playing an infinite recursive game of random tic-tac-toe. Using the new Block primitive to manage shared state of multiple "Board" blocks (one for each game) between "nought" and "crosses" subflows. New games are started again by recursively calling the parent "tic_tac_toe" flow. Each move / square is a Prefect task. Rules: • Loser gets the starting move in the next game • Draw spawns two new games Subflow states: • Win = Completed • Lose = Failed • Draw = Cancelled You can keep track of the game tally between X and O by filtering flow run states. Open questions: • Are blocks threadsafe? Gist: https://gist.github.com/topher-lo/d2cf1124f68371716725869917d31aaf
    🎉 9
    🙌 6
    😂 7
    😮 4
    :its-beautiful: 6
    😒piderman-pointing: 4
    c
    j
    a
    • 4
    • 6
  • n

    Nico Neumann

    08/03/2022, 2:57 PM
    Code Contest I am currently exploring the prefect 2.0 features (blocks, async, deployments, ui, schedules, etc.) and thought it would be interesting to do it in a fun way. This flow fetches random quotes from the internet using async/await and combines them with a random cat gif. The result is then send to a Slack channel using Prefect Slack Block. The user can then vote which one they like the most, of course counting starts at 0. The flow is scheduled to run every day through Prefect schedules. Currently it is only possible to send text to Slack, that’s why I provided a pull request which allows to send e.g. images or gifs using Slack blocks: https://github.com/PrefectHQ/prefect/pull/6200
    🐱 4
    :catjam: 3
    🎉 6
    :party-parrot: 5
    👍 3
    :thank-you: 2
    🙌 3
    c
    • 2
    • 2
  • k

    Khuyen Tran

    08/04/2022, 9:47 PM
    I just created

    a video▾

    to show a use case of deployment in Prefect 2.0: Schedule your data workflow. This video covers the main concepts such as deployment, work-queue, and agent. If you are new to deployment, this video should make it clearer to you.
    📺 1
    🙌 2
    🔥 8
    :party-parrot: 6
    👍 1
    :thank-you: 1
    💯 1
    f
    • 2
    • 2
  • c

    Cole Murray

    08/05/2022, 3:13 AM
    Code Contest At Empiric, we’re a IOT platform for industrial manufacturing. We utilize prefect as our distributed job scheduler. Our infrastructure utilizes AWS ECS, running prefect 2.0. The system resources are controlled via AWS CDK, making it easy for teams to launch their own stack of job orchestration system, or run on the central company infra. Below, one of our flows is responsible for performing vibration analysis on sensor data from machines out in the field. The flow can process multiple days of data from multiple sensors, and utilizes sub-flows for each individual sensor & day combination. For tasks, we abstract the logic into individual classes, allowing for a decoupling of the logic from underlying prefect tasks. This simplifies testing as well, as our unit-tests have no prefect related dependencies.
    👍 5
    🙌 2
    🚀 3
    🔥 2
    ☝️ 3
    ❤️ 2
    🎉 5
    :prefect: 2
    :party-parrot: 2
    a
    • 2
    • 1
  • v

    Viet Nguyen

    08/05/2022, 8:21 AM
    I like watching the flow running with this feature, looks like a spider's web:
    🙌 2
    :radar: 7
    🚀 3
    🕷️ 1
    😀 1
    🕸️ 8
  • a

    Alix Cook

    08/05/2022, 7:36 PM
    made a simple webhook client to transform prefect webhook events into datadog events (also a shameless plug to maybe support generic webhooks in prefect 2.0 🙏 ) https://github.com/acookin/prefect-dd-webhook I've been using this to generate flow run metrics, and to alert when flows are not running as expected
    🙌 2
    :thank-you: 2
    🚀 2
    a
    • 2
    • 2
  • k

    Khuyen Tran

    08/08/2022, 2:15 PM
    I just published the article Detect Defects in a Data Pipeline Early with Validation and Notifications. This article covers how to: • Use Deepchecks to validate components in the research phase of your data science pipeline • Use Prefect to send notifications when a validation failed Source code.
    🎉 5
    :party-parrot: 4
    🚀 3
    ❤️ 2
    :prefect: 2
    a
    j
    • 3
    • 3
  • f

    Falk

    08/16/2022, 1:29 PM
    I wrote a really small and compact guide on how to automatically trigger Jira Tickets on Task Failures: https://dev.to/codiefz/create-jira-ticket-on-prefect-task-failure-cj3 Following the docs I ran into some issues, mainly because some imports were missing, so perhaps it might be useful to some
    🎉 4
    🚀 3
    👍 3
    👀 1
    :thank-you: 2
    🙌 3
    j
    • 2
    • 1
  • s

    Saad Zaheer

    08/18/2022, 4:01 PM
    We wrote about how we are using Prefect to orchestrate all the components and data pipelines in Endeavor's home grown machine learning platform. https://medium.com/@endeavordata/machine-learning-platform-at-endeavor-93ba88b66986 Would love to hear how others have used Prefect in similar ML use cases!
    :prefect: 6
    🚀 9
    🙌 10
    :thank-you: 7
    💯 7
    a
    j
    • 3
    • 2
  • a

    ale

    08/19/2022, 7:17 AM
    Hey folks 👋 I’ve just released
    prefect_transfrom==0.1.6
    . Docs che be found here. This release introduces a new Transform Block, which is now required to handle sensitive information to authenticate on Transform. Give it a try! Thanks @Andrew Huang for the guidance on implementing Prefect Block! 🙌
    :marvin-duck: 9
    💯 6
    👍 4
    :thank-you: 4
    🙌 2
    🚀 10
  • a

    ale

    08/19/2022, 2:00 PM
    And
    prefect_stitch==0.1.3
    has a brand new block too!
    StitchCredentials
    , which can be used to handle Stitch access token by leveraging the Prefect Block concept 😛arty-parrot: 😛arty-parrot: 😛arty-parrot: Docs can be found here.
    🙌 8
    ✅ 5
    :thank-you: 7
    👍 8
    🔥 6
  • j

    Jeremiah

    08/19/2022, 7:27 PM
    @ale love how you’re already putting blocks to great use!
    🎉 5
    🙌 1
  • c

    Chris L.

    08/21/2022, 9:32 AM
    Hello Prefect community. 👋 Wanted to share a small code snippet I've been using to resolve this open issue: https://github.com/PrefectHQ/prefect/issues/5853 It's not technically a "fix" as I am creating new functions in the namespace via currying 🍛, but I think it's a cleaner solution compared to the current hack of using "deepcopy". @Jacob Danovitch @Jeff Carrico @Anna Geller Hope this helps others encountering this issue!
    import asyncio
    from prefect import task, flow
    
    @task
    async def print_x(x):
      print(x)
      await asyncio.sleep(1)
    
    def build_subflow(name):
      @flow(name=f"subflow:{name}")
      async def subflow(x):
        await print_x(x)
      return subflow
    
    @flow
    async def parent_flow():
      futures = [build_subflow(name=x)(x) for x in ["a", "b", "c"]]
      await asyncio.gather(*futures)
    :gratitude-thank-you: 1
    a
    k
    • 3
    • 6
  • a

    Anna Geller

    08/22/2022, 6:46 PM
    If you like Airbyte, check out @Nate's latest blog post showing how to use our
    prefect-airbyte
    collection in Prefect 2.0
    ✈️ 7
    🙌 11
  • k

    Khuyen Tran

    08/23/2022, 8:41 PM
    For those who just started with 2.0,

    this video▾

    explains what a flow is and how to use a flow in your project.
    💥 5
    🙌 2
  • r

    Ryan Peden

    08/24/2022, 3:43 AM
    For anyone looking to get up and running with Prefect on Azure, I've published two tutorials over the past week (with more on the way!): • How to Run a Prefect 2 Agent in an Azure Container Instance • How to create Azure Blob Storage event-driven Prefect 2 flows with Azure Functions I also published a quick tutorial on how to run a Prefect agent as a systemd service on Linux.
    🚀 2
    :cloud2: 2
    :thank-you: 4
    💯 2
    🙌 4
    :azure: 3
    :prefect: 2
    o
    j
    • 3
    • 4
  • j

    Jeff Hale

    08/25/2022, 4:39 PM
    If you’re looking to migrate your Databricks tasks from Prefect 1 to Prefect 2 with the prefect-dbt collection, check out this hot-off-the-press guide from @Andrew Huang! 🔥
    :thank-you: 2
    :databricks: 4
    🙌 2
  • p

    Pranit

    08/26/2022, 6:03 AM
    Hello I am trying to integrate prefect cloud with Airbyte. But I am facing below error
    ModuleNotFoundError: No module named 'prefect.tasks.airbyte'; 'prefect.tasks' is not a package
    Have I missed to install any dependency ? I didnt see anything on airbyte/prefect pages.
    ✅ 1
    j
    a
    • 3
    • 3
  • j

    Jeremiah

    08/26/2022, 6:48 PM
    Elevating this so it doesn’t get buried — thanks @Adebayo Adejare! https://prefect-community.slack.com/archives/CL09KTZPX/p1661533329261399?thread_ts=1661493823.803219&cid=CL09KTZPX
    🚀 3
    🙌 9
    :thank-you: 5
    :party-parrot: 4
  • r

    Ryan Peden

    08/30/2022, 4:19 PM
    I noticed a few community members trying to run Prefect 2 on CentOS 7, which can be tricky because it comes with an old version of Python and a very old version of SQLite. CentOS 7 is tricky because it is old but still supported, so many enterprises still use it and consequently, you might need to run Prefect on it. So, I put together a quick tutorial that demonstrates a couple of ways to get Prefect up and running quickly on CentOS 7 (and Red Hat Enterprise Linux 7, since CentOS 7 is built from RHEL 7's codebase.)
    :party-parrot: 6
    👍 5
    :thank-you: 3
    👏 3
    😛anda-dancing: 3
    💯 3
    🚀 4
    ❤️ 4
    🙌 5
    :blob-attention-gif: 4
    :marvin: 3
  • k

    Khuyen Tran

    08/31/2022, 3:47 PM
    What is a subflow? When should you use subflows and when should you use tasks?

    This short video▾

    will answer those questions.
    :thank-you: 3
    🚀 3
    :hattip: 1
    :blob-attention-gif: 1
    s
    c
    • 3
    • 3
  • j

    Jeff Hale

    09/01/2022, 1:21 PM
    Hey all! 👋 I demo my two favorite new Prefect 2.3 features in the video linked here. Hope you find it helpful. 🙂
    🙌 3
    :thank-you: 1
    ❤️ 3
    :upvote: 1
  • j

    Jeff Hale

    09/01/2022, 7:10 PM
    I wrote a guide to making deployments with a Python file instead of the CLI. Check out this post and please let me know if you find it helpful! 🐍 https://medium.com/the-prefect-blog/deploy-prefect-pipelines-with-python-perfect-68c944a3a89f
    🚀 8
    :thank-you: 4
    🙌 1
    💯 3
    :prefect: 3
    :marvin-duck: 4
  • a

    Alberto de Santos

    09/05/2022, 3:56 PM
    Hi everyone! I'd like to share with you a nice project orchestrated entirely with Prefect on top of Raspberry Pi. The project consists of an ML/AI-based football/score predictive model, running predictions for 100+ competitions across the globe, covering the entire Machine Learning end-to-end: from data acquisition, validation, model prediction/training and evaluation. I have also used the integrations with Slack (modified for Telegram) and Great Expectations for data validation/quality. In case you are interested, here is the link for the Telegram channel: t.me/alea_jacta_est_circe if you want to have a look to the predictions and results done on a daily basis with Prefect as the orchestrator engine. Hope you like it and enjoy it as much as I did integration Prefect with Raspberry Pi!
    ⚽ 1
    🥧 4
    🎉 5
    🙌 10
    🏈 5
    a
    c
    l
    • 4
    • 5
  • a

    Anna Geller

    09/06/2022, 2:29 PM
    Just published - if you have some workflows that depend on events, such as when a new file arrives in a certain directory, this post may help you design your flow
    🙌 5
    :party-parrot: 6
    🎉 4
    :prefect-duck: 4
  • h

    Henning Holgersen

    09/07/2022, 9:57 AM
    A few days ago I asked about prefect+meltano in docker, and although it seems it wasn’t a common use case, I now have a working prototype if anyone is interested: https://github.com/radbrt/prefect_flows/tree/main/projects/jottings/meltano_flow It is in prefect 1 for now, but should be easy to port.
    :thank-you: 5
    :prefect: 5
    🙌 5
    a
    r
    • 3
    • 2
  • a

    Anna Geller

    09/07/2022, 9:17 PM
    if you couldn't join today's PrefectLive, I highly recommend watching the recording

    here▾

    👀 3
    🙏 2
    👍 2
Powered by Linen
Title
a

Anna Geller

09/07/2022, 9:17 PM
if you couldn't join today's PrefectLive, I highly recommend watching the recording

here▾

👀 3
🙏 2
👍 2
View count: 1