https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
show-us-what-you-got
  • k

    Khuyen Tran

    09/08/2022, 7:02 PM
    The latest versions of Prefect make it easy to use GitHub as the storage and Docker as the infrastructure using blocks. In my article Create Robust Data Pipelines with Prefect, Docker, and GitHub, you will learn how to use Prefect to easily store your flow in GitHub and execute it in a Docker container.
    🙌 3
    ✅ 3
    :marvin-duck: 3
    👍 8
    :thank-you: 2
    :prefect: 1
    💙 1
    🎉 6
    🚀 4
    t
    • 2
    • 2
  • a

    Alix Cook

    09/08/2022, 11:02 PM
    wrote up a little gist around how I was able to use blocks to allow for cleaning up and managing flow runs in a restrictive environment (and was able to solve the problem i described in this thread) https://gist.github.com/acookin/aa5f0d718997e7a645cbc160e2c94a16
    :thank-you: 8
    🙌 5
    👏 4
    :party-parrot: 3
    e
    a
    • 3
    • 2
  • a

    Anna Geller

    09/14/2022, 8:30 PM
    Today we published: • 2.4.0 release blog post • @Oscar Björhn joined us on Prefect Live - you can

    watch the recording▾

    if you missed it!
    Thanks to Oscar for being such a great community member and for joining us today 💙
    🚀 2
    🎉 1
    :prefect-duck: 3
    🙌 2
    👍 1
  • s

    Saurabh Indoria

    09/15/2022, 4:21 AM
    Checkout how Quilt.AI used Prefect to empower its scalable & flexible ML & Data platform. https://www.prefect.io/guide/case-studies/prefect-quilt-ai/
    🙌 6
    👏 4
    :cool-llama: 3
    👍 3
    🚀 5
    :thank-you: 3
  • b

    Bianca Hoch

    09/15/2022, 9:23 PM
    Posting here so that it doesn't get buried in community - Thank you @Hallen Maia for your excellent work!
    :party-parrot: 3
    👏 7
    🎉 5
  • e

    Erik

    09/17/2022, 10:52 PM
    Hi all! This is a really simple one. I’m brand new to the platform, just a few days in. I already get too many Slack notifications, so I prefer to keep app alerts/failures/exceptions in Google Chat. That way I know not to ignore it like my other slack notifications 😂 jk. It turns out the google chat JSON structure for simple text works exactly the same as the slack one, which means you can simply paste in the google chat webhook in place of the
    slack webhook
    for block/notifications in the prefect ui. If you haven’t used google chat webhooks, it’s even slightly easier than slack. https://developers.google.com/chat/how-tos/webhooks#create_a_webhook
    👍 1
    :thank-you: 2
    🙌 2
    :upvote: 2
    a
    • 2
    • 1
  • a

    Anna Geller

    09/19/2022, 12:31 PM
    If you want to read more about: • the difference between tasks, flows, and subflows • when to return data 📊, state ✅, or Prefect future 🔮 then add this post to your reading list for the week: https://medium.com/p/5eaabdfbb70e Have a great start to the new week!
    🙌 8
    ⭐ 3
    🔥 7
    :blob-attention-gif: 2
    🚀 5
    👀 4
    o
    a
    • 3
    • 6
  • a

    Anna Geller

    09/20/2022, 9:17 AM
    New blog post by @justabill 🔥
    :thank-you: 7
    :blob-attention-gif: 5
    🎉 4
    :party-parrot: 6
  • b

    Benjamin Cerigo

    09/21/2022, 12:37 PM
    If any one is interested in how you can: 1. Define Deployments in a separate repo from flows 2. Define Deployments that run any docker container and command (not just prefect flows) Here is an [experimental repo](https://github.com/datavaluepeople/prefect_deployment_experiment) that shows how. Thanks @Anna Geller and @Khuyen Tran for the support on this.
    :thank-you: 5
    🚀 5
    🎉 8
    m
    • 2
    • 2
  • r

    Rich Tata

    09/22/2022, 1:21 AM
    Prefect sold out, smh. But seriously, what are the chances of getting this flow run name?
    🦞 20
    🤣 16
    😂 4
    a
    c
    • 3
    • 2
  • a

    Anna Geller

    09/27/2022, 3:24 PM
    Scheduled serverless flows in 5 minutes - this post shows how you can use alternative tools (e.g., GitHub Actions workflow or legacy orchestrators) to run Prefect flows on schedule
    👍 4
    :party-parrot: 3
    :blob-attention-gif: 7
    🎉 4
    ✨ 3
  • b

    badasstronaut

    09/29/2022, 6:51 PM
    Can I get feedback/corrections on this sequence diagram? I’m using
    prefect-aws
    and
    prefect-dask
    +
    dask-cloudprovider
    for autoscaling
    👀 1
    t
    b
    +5
    • 8
    • 38
  • d

    Daniel Conway

    10/06/2022, 6:57 PM
    from prefect import flow, task from prefect import flow, task import yfinance as yf from time import sleep from random import randint from pandas import DataFrame from datetime import timedelta @task(retries = 5, retry_delay_seconds = 5) def getData(ticker): print('before: ', ticker) df = yf.download(ticker) print('after ') return df @task def displayTicker(df:DataFrame): print(df) @flow def getTickerData(symbols): print('in getTickerData') dfs = getData.map(symbols) print('After getTickerData') #displayTicker(dfs) return dfs if name == "__main__": print(' --------------------------------------- ') print(' --------------------------------------- ') ticker = ['DE','GE','IBM','CSCO'] stockTable = getTickerData(ticker) print ('*** ', ticker, ' * ') print (stockTable) print(' --------------------------------------- ') print(' --------------------------------------- ')
    :marvin-duck: 1
    🚀 1
    a
    • 2
    • 1
  • a

    Anna Geller

    10/07/2022, 10:53 AM
    Shoutout to @Oscar Björhn for giving excellent product feedback that was directly used to build a new feature. The
    run_deployment
    utility was driven, a.o., by his feedback in the Community and

    this live stream▾

    -- the timestamp in the video shows the reason why Oscar implemented his own way of running flows from deployment, and the final implementation in Prefect 2.5 even used the same name as the one implemented by Oscar! 🚀 Great work, and really glad to have you in the community! 💙
    👍 5
    🎉 7
  • a

    Anna Geller

    10/11/2022, 11:09 AM
    How to build a data platform with Prefect 😛refect:*+ dbt* :dbt: + Snowflake ❄️ - part 1 (conceptual post)
    🎉 10
    :party-parrot: 3
    ❄️ 4
    :dbt: 3
    s
    m
    • 3
    • 9
  • k

    Khuyen Tran

    10/14/2022, 2:14 PM
    My article on Create a Maintainable Data Pipeline with Prefect and DVC Example project.
    💯 9
    🔥 4
    :party-parrot: 3
    🙌 5
    :blob-attention-gif: 2
  • h

    Henning Holgersen

    10/16/2022, 2:02 PM
    I have pondered deployment patterns in Prefect 2 a little more, and come up with something I might like: https://github.com/radbrt/orion_flows Hopefully the readme is understandable, but the gist of it is to create deployments in Python files (the
    Deployment
    class), in separate files that imports the flow and creates+applies a deployment. These files have a common naming pattern, all ending with
    .deployment.py
    , so that github action can pick them up and run them. This way, the flow file itself remains clean, and we don’t hardcode any config in the github action yaml, making it fully configurable. Thanks to @Emil Christensen for explaining the point of deployments to me in a way I finally understood. Oh, and there is also a cookiecutter template to create a new skeleton flow.
    🙌 10
    :marvin: 5
    :thank-you: 2
    👀 1
    e
    r
    +2
    • 5
    • 15
  • m

    Michael Adkins

    10/19/2022, 4:57 PM
    👋 I just released https://github.com/madkinsz/prefect-kv which is a key-value store for use in your flows/tasks built on top of Prefect’s JSON block!
    🔑 2
    🙌 1
    🔥 11
    :blob-attention-gif: 10
    🚀 3
  • k

    Kalise Richmond

    10/19/2022, 7:03 PM
    Hey all, huge shout out to @Serina for revamping our Prefect Recipes repo! 🧑‍🍳 There's several recipes to help getting started deploying agents on different infrastructure such as k8s or docker containers. If you haven't checked it out yet, it's got lots of great examples and is continuing to grow. It's a great spot to contribute with big ideas 💡
    :blob-attention-gif: 6
    :thank-you: 2
    🤩 3
    🎉 10
    🚀 8
    :prefect-duck: 5
    :marvin: 7
    🔥 5
    👍 2
    💡 2
    🙌 1
    b
    • 2
    • 3
  • j

    Jeff Hale

    10/19/2022, 7:39 PM
    I wrote a guide to help folks get started contributing to open source in Towards Data Science! If you haven’t tried before, it’s easier than you might think. At the end of the post you’ll see a sneak peak of Prefect’s #hactoberfest! 🎃
    🔥 9
    🙌 7
    👍 2
    :thank-you: 2
    🍁 6
    🍂 7
    🚀 3
  • c

    Chris Reuter

    10/20/2022, 1:38 PM
    If you're looking for a good way to explain what Prefect is to your boss, co-worker, significant other, grandparent, neighbor or that person in the grocery - we made this for you:

    https://youtu.be/ZK1s8OfVSpY▾

    ✈️ 13
    :blob-attention-gif: 2
    🙌 6
    🚀 3
    :party-parrot: 6
    s
    • 2
    • 3
  • r

    Ryan Peden

    10/21/2022, 4:11 AM
    As promised in Anna's release Prefect 2.6.4 release announcement, I just published a post introducing the new Azure Container Instances infrastructure block and showing how to set it up to run your flows: https://medium.com/the-prefect-blog/serverless-prefect-flows-with-azure-container-instances-f2442ebc9343
    :blob-attention-gif: 6
    🙌 9
    🔥 6
    ❤️ 5
    💯 9
    :azure: 5
    :thank-you: 4
    👏 4
    :prefect: 5
    o
    j
    +3
    • 6
    • 18
  • f

    Francesco Bartoli

    10/27/2022, 10:39 AM
    I’ve shared parts of the code that I have implemented to orchestrate an ingestion of spatial data datasets which need to be paused for approval https://discourse.prefect.io/t/implementing-waiting-for-approval-in-prefect-2/1558/6?u=francbartoli. I liked to apply the workaround suggested by @Anna Geller which is working nicely but also I’d love to have this feature handled natively in Orion. Very soon I will share the whole code publicly
    💯 1
    :gratitude-thank-you: 2
    💪 1
    👏 1
    :prefect: 1
    🚀 1
    🙌 3
  • k

    Khuyen Tran

    10/31/2022, 3:11 PM
    If you want to send an alert when a flow fails by adding only one decorator, use prefect-alert.
    from prefect import flow, task 
    from prefect.blocks.notifications import SlackWebhook
    from prefect_alert import alert_on_failure
    
    @task
    def may_fail():
        raise ValueError()
    
    @alert_on_failure(block_type=SlackWebhook, block_name="test")
    @flow
    def failed_flow():
        res = may_fail()
        return res
    
    if __name__=="__main__":
        failed_flow()
    Thanks @Ryan Peden for contributing the collection!
    :gratitude-thank-you: 1
    👍 1
    ⭐ 2
    💯 1
    🙌 8
    🎉 4
    🔥 7
  • k

    Khuyen Tran

    11/01/2022, 4:12 PM
    Have you ever wanted to automatically rerun modified components of a data pipeline? In this article, you will learn how to: • Use GitHub Actions to run a workflow when you push a commit • Use DVC to run stages with modified dependencies Code used in the article.
    :marvin: 6
    🙌 4
    :party-parrot: 2
    🎉 4
    :upvote: 1
  • r

    Ryan Peden

    11/03/2022, 2:49 PM
    If you use Prefect on Google Cloud, you might enjoy this post I wrote about how to run Prefect flows as Google Cloud Run jobs: https://medium.com/the-prefect-blog/serverless-prefect-flows-with-google-cloud-run-jobs-23edbf371175
    🙌 5
    🎉 4
    :gratitude-thank-you: 2
    🚀 4
    :marvin: 3
    🔥 3
    👀 3
    f
    • 2
    • 3
  • b

    Ben Muller

    11/09/2022, 10:05 PM
    Hey all - I seem to get unlimited help from all of the incredible prefect peeps on the #prefect-community channel, so thought that in some way/shape/form may be useful to others. It is very much specific to my use case, but maybe there are pieces that others can use. It is a CI pattern I am using for a repository of ECSTask flows that run across different workspaces in Prefect 2.0. The scripts might help out others. Have a look and feel free to shoot me any questions. https://github.com/bennnym/prefect-ci-pattern
    🙌🏽 1
    👍🏽 1
    🙌 14
    🚀 8
    👍 7
    :thank-you: 7
    :upvote: 4
    :gratitude-thank-you: 5
    🎉 6
    ❤️ 4
    a
    t
    m
    • 4
    • 6
  • c

    Carlo

    11/10/2022, 2:53 PM
    This is similar to how we have organized things too, except we just kept a list in python over a yaml config. Thanks for sharing.
    🙌 2
  • s

    Stéphan Taljaard

    11/16/2022, 9:19 AM
    This is so simple, yet quite amazing:
    from typing import Literal
    from prefect import flow
    
    AllowedStrings = Literal["A", "B"]
    
    @flow
    def example(str_input: AllowedStrings):
        print(str_input)
    
    example("C")
    --->
    11:15:58.598 | INFO    | prefect.engine - Created flow run 'blond-auk' for flow 'example'
    11:15:58.598 | ERROR   | Flow run 'blond-auk' - Validation of flow parameters failed with error: ParameterTypeError: Flow run received invalid parameters:
     - str_input: unexpected value; permitted: 'A', 'B'
    🙏 1
    :its-beautiful: 1
  • a

    Anna Geller

    11/16/2022, 1:05 PM
    cross-sharing from @Eden - post on medium :kubernetes: :gcp:
    :prefect-duck: 5
    🙌 4
    :blob-attention-gif: 2
    :marvin: 1
    c
    • 2
    • 1
Powered by Linen
Title
a

Anna Geller

11/16/2022, 1:05 PM
cross-sharing from @Eden - post on medium :kubernetes: :gcp:
:prefect-duck: 5
🙌 4
:blob-attention-gif: 2
:marvin: 1
c

Chris Reuter

11/16/2022, 2:20 PM
and if you're interested in anything GCP related, don't forget to join #prefect-gcp!
👍 2
View count: 3