https://prefect.io logo
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
show-us-what-you-got
  • a

    Anna Geller

    05/19/2022, 5:34 PM
    I genuinely don't know how @Jeremiah finds time for creating content next to being a CEO, a dad, a Twitter 🌠 and very actively contributing to the codebase and product, but here are two recent pieces of content about dataflow and the problem of orchestration created by Jeremiah: β€’ Data Exchange - podcast β€’ Negative Engineering and the Art of Failing Successfully - blog post This might be especially interesting to those who want to find out more about the problem Prefect tries to solve and the product philosophy behind it. And why you should never say "just".
    :marvin: 1
    πŸ’― 10
    πŸ™Œ 5
    j
    a
    • 3
    • 2
  • c

    Chris Reuter

    05/26/2022, 7:03 PM
    Thank to @Nate for writing a cool (and hilarious) guide demonstrating how to turn your existing Python code into scheduled, repeatable, Prefect flows. Check it out! https://www.prefect.io/blog/your-python-job-is-already-a-prefect-flow-it-just-doesnt-know-it-yet/
    πŸ‘ 1
    :marvin: 9
    πŸ’ͺ🏽 1
    t
    n
    • 3
    • 3
  • l

    loris

    05/31/2022, 8:08 PM
    Knowledge management sounds cool but it rarely works. Why? I recently had a great conversation with Stephanie Barnes the creator of Radical Knowledge Management. Stephanie advocates for curiosity, playfulness and creative learning as the cornerstones of the journey towards knowledge management, principles that (as she says) have been educated out of us through production-line thinking. Read / ListenπŸ‘‡πŸ» https://www.discoveringdata.com/podcast/034-creative-learning
    a
    • 2
    • 1
  • a

    Anna Geller

    05/31/2022, 8:54 PM
    set the channel topic: Built something cool with Prefect? Wrote an article or a social media post featuring Prefect? Show us what you got!
    ❀️ 1
  • a

    Anna Geller

    05/31/2022, 8:55 PM
    set the channel description: A place for sharing knowledge about Prefect, tips and tricks about working with data, and best practices about workflow orchestration and working with Python
  • a

    ale

    06/08/2022, 7:29 AM
    Hey folks πŸ‘‹ Here you can find a blog post from the awesome @Stefano Cascavilla explaining how we do Data Engineering at Cloud Academy! https://cloudacademy.com/blog/how-do-we-transform-and-model-data-at-cloud-academy/
    πŸ™ 2
    :prefect: 5
    πŸ™Œ 3
    πŸ”₯ 5
    a
    k
    • 3
    • 3
  • c

    Chris Reuter

    06/08/2022, 10:13 PM
    Also check out @Madison Schott’s post from yesterday about using πŸ˜›refect:, :dbt: and πŸ˜’lack: to monitor data freshness: https://towardsdatascience.com/how-fresh-are-your-data-sources-e8db53cf4653
    🍊 8
  • c

    Chris Reuter

    06/09/2022, 1:00 PM
    A fresh quick start tutorial is out today! Check out Getting Started with Prefect 2.0 by @Peyton Runyan for a step-by-step tutorial for integrating Prefect into your code.Peyton's walkthrough is an agile method for adopting Prefect quickly with just a few lines of code and then expands to scheduling, work queues and agents. Want to write your own Prefect content? It's easy! If you have any questions, we're here to help. Just post right in this channel.
    ❀️ 8
    πŸ’― 6
    :wizard2: 5
    πŸ‘ 5
  • c

    Chris Reuter

    06/10/2022, 5:56 PM
    You may have noticed our recently updated resources section on our website, appropriately titled The Guide! It features community content, tutorials, guides, company news, videos, podcasts and more. Enjoy exploring, and we hope that you find it useful during your Prefect Journey. https://www.prefect.io/guide/ Also check out our clutch Zelda reference on our announcement tweet ❀️
    :marvin: 6
  • s

    Slackbot

    06/12/2022, 8:41 AM
    This message was deleted.
  • s

    Simon SpΓ€ti

    06/14/2022, 5:41 PM
    I wrote about the trends in data orchestrations, and of course, prefect is a part of it πŸ™‚ Hope it's helpful, and looking forward to your comments. I'm covering topics such as: β€’ The shift from Data Pipelines to Data Products β€’ Data-aware Pipelines that know about the inner life of a task β€’ Why a declarative approach with higher-level abstractions helps β€’ How abstractions improve reuse code between complex cloud environments β€’ Where do we come from with the evolution of Data Pipeline Orchestration β€’ Open-source data orchestration tools such as Airflow, Prefect, Dagster https://airbyte.com/blog/data-orchestration-trends
    πŸ‘ 2
    πŸ™‚ 9
    k
    k
    s
    • 4
    • 8
  • k

    Khuyen Tran

    06/28/2022, 4:00 PM
    Have you ever wondered what negative engineering is? I created

    this videoβ–Ύ

    to clarify the difference between negative engineering and positive engineering.
    πŸ‘ 3
    πŸ™Œ 9
  • k

    Khuyen Tran

    07/01/2022, 3:59 PM
    I just posted about

    adding observability to your Python script with Prefectβ–Ύ

    . This video meant to be a short and fun watch
    πŸ™Œ 12
    πŸŽ‰ 3
    r
    b
    • 3
    • 2
  • k

    Khuyen Tran

    07/05/2022, 4:23 PM
    I just posted

    a short and intuitive introduction to caching Python function in Prefectβ–Ύ

    in case you want to wrap your head around caching
    πŸ™Œ 4
    πŸŽ‰ 4
    r
    • 2
    • 2
  • k

    Khuyen Tran

    07/06/2022, 3:28 PM
    I often use GitHub feed to get updated with cool and trending libraries. However, I don’t care about 70% of what I saw on my feed so I built a workflow to get the data from GitHub and create a dashboard. View my article on this project here.
    πŸš€ 6
    :marvin: 5
    πŸ™Œ 4
  • k

    Khuyen Tran

    07/07/2022, 4:12 PM
    I just posted a

    short video explaining what retries isβ–Ύ

    and how to use retries with Prefect
    πŸ‘ 5
    πŸš€ 8
  • k

    Kevin Focke

    07/10/2022, 2:38 PM
    Why do some financial professionals like to "make it rain" πŸ’¦β˜”? I made a program to figure that out. My program generates spurious stock correlations between stock price and rainfall based on historical movers & shakers. Obviously not investment advice. Github: https://github.com/KevinFocke/noisy_stocks_data_orchestrator Website: https://www.noisystocks.com/
    πŸ’ͺ 5
    β˜” 8
    a
    c
    k
    • 4
    • 4
  • k

    Khuyen Tran

    07/11/2022, 4:38 PM
    Do you know that you can add flows under another flow in Prefect 2.0?

    This videoβ–Ύ

    will explain the difference between task, flow, subflow, and how to create a subflow.
    πŸ™Œ 8
    :party-parrot: 7
    r
    • 2
    • 6
  • j

    Joshua Greenhalgh

    07/11/2022, 5:28 PM
    Dunno if useful for anyone but I wanted a quick view of the states of my flows so put together this CLI script
    πŸ‘€ 4
    k
    a
    • 3
    • 12
  • o

    Oliver Mannion

    07/12/2022, 10:27 AM
    Some Prefect Orion examples running self-contained in a local kubernetes cluster. Flows β€’ Dask flow using a dask local cluster β€’ Dask kubes flow using an ephemeral dask cluster on Kubernetes β€’ Parameterized flow using a custom Docker image containing additional modules β€’ Ray flow that runs on an existing ray cluster β€’ Sub flow that is trigger by a parent flow Deployments to Kubernetes using β€’ the default OrionPackager to store the flow source in Orion β€’ the default OrionPackager to store the flow import path in Orion β€’ the FilePackager to store the flow source in S3 (minio) Batteries (mostly) included. πŸ”‹
    ❀️ 10
    :thank-you: 7
    πŸ™ŒπŸΌ 1
    :oh-yeah: 3
    πŸŽ‰ 3
    :upvote: 7
    πŸ™Œ 12
    a
    m
    • 3
    • 3
  • s

    Slackbot

    07/19/2022, 2:25 PM
    This message was deleted.
    k
    • 2
    • 1
  • k

    Khuyen Tran

    07/20/2022, 10:49 PM
    Did you know that you can now send Slack notifications based on the state of your flow by simply clicking on the UI in Prefect 2.0b8 and above? This article shows how you can do exactly that.
    :party-parrot: 2
    πŸš€ 8
    ❀️ 13
    m
    • 2
    • 6
  • a

    Anna Geller

    07/25/2022, 2:54 PM
    Many of you requested a real-time streaming recipe - it's finally released!
    :marvin: 5
    πŸ™Œ 11
    πŸ‘€ 4
    ❀️ 12
    πŸ₯³ 4
    :party-parrot: 8
    :hattip: 4
    n
    m
    • 3
    • 3
  • v

    Valeria Romero

    07/25/2022, 7:11 PM
    Hey, everyone! We constructed a simple Prefect-1.0 AWS ECS Boilerplate to deploy multiple flows with dependencies using GitHub Storage and ECSRun. We set up the ECS infrastructure with Terraform; with GitHub Actions we build and push a Docker image to ECR and register the flows to Prefect; and, finally, we run the flows on Prefect Cloud UI with a local ECS Agent. This boilerplate facilitates: - Import from flow modules in different directories - Automatic Prefect flow configuration and registration - The use of GitHub Secrets and dotenv variables - ECS Cluster setup with Terraform - The connection to ECR and Prefect using GitHub Actions Check out the repository here: https://github.com/eidos-ai/prefect-aws-boilerplate. We hope this template is useful to others, and we look forward to reading your feedback!
    πŸ’― 2
    πŸš€ 1
    πŸ™Œ 5
    πŸ™ŒπŸ» 1
    :thank-you: 3
    a
    • 2
    • 2
  • a

    Anna Geller

    07/27/2022, 9:26 PM
    @George Coyne has recently published this great blog post to help you migrate your Prefect 1.0 flows to Prefect 2.0 :cloud2:
    :thank-you: 5
    :prefect: 4
    😍 2
    a
    • 2
    • 1
  • b

    Benny Warlick

    07/31/2022, 1:21 AM
    Hey all, I created a simple example showing how to deploy a Prefect agent to a docker on Google Cloud Compute Engine, using Github Actions. Flow stored in Google Cloud Storage. I had previously posted this when Prefect 2 was in beta, and just updated it for 2.0.1. https://github.com/BennyJW/prefect2-docker-gce
    πŸ‘ 2
    πŸŽ‰ 9
    πŸ™Œ 8
    :thank-you: 3
    :marvin: 2
    a
    k
    • 3
    • 8
  • c

    CA Lee

    07/31/2022, 5:02 AM
    Hey everyone, I would like to share a simple pattern for single deployment file to execute multiple flows on Prefect Orion 2.0.1: https://www.notion.so/jlom/Prefect-Orion-Deployment-Guide-4cc9393bd4c943a38777a92027522cf8
    πŸ‘ 2
    πŸ™Œ 9
    ❀️ 5
    :thank-you: 2
    :party-parrot: 2
    πŸ‘πŸ» 1
    k
    j
    j
    • 4
    • 3
  • k

    Khuyen Tran

    07/31/2022, 4:50 PM
    For those who want to send notifications in 2.0, check out this article. I just updated the code so that it will work with the latest version of 2.0.
    :thank-you: 2
    πŸ™Œ 4
  • c

    Chris Reuter

    07/31/2022, 5:35 PM
    Loving all of the community contributed content! We'll add these to The Guide and would encourage anyone to post their templates/patterns/examples here.
  • a

    Andreas

    08/01/2022, 2:35 PM
    Code Contest This is a simple flow that downloads videos of a playlist in parallel using pytube
    #!/usr/bin/env python
    import argparse
    from pytube import Playlist, YouTube
    from prefect import task, flow
    from pydantic import HttpUrl
    from prefect_dask import DaskTaskRunner
    
    @task(retries=3, retry_delay_seconds=4)
    def download_video(video: YouTube):
        video.streams.get_highest_resolution().download()
        return
    
    @flow(task_runner=DaskTaskRunner)
    def playlist_downloader(playlist_url: HttpUrl):
        p = Playlist(playlist_url)
        for video in p.videos:
            download_video.submit(video)
        return
    
    
    if __name__ == "__main__":
        parser = argparse.ArgumentParser(description='Downloads all youtubes videos in a playlist in parallel using the highest available resolution')
        parser.add_argument('playlist', help='a playlist url to download videos')
        
        args = parser.parse_args()
        playlist_downloader(args.playlist)
    It is really great that using Prefect we are able to orchestrate this functionality in so few lines of code and use features like running stuff in parallel using Dask and having retry functionality πŸ˜›refect-duck:
    :prefect2: 5
    :marvin: 4
    πŸ™Œ 9
    :party-parrot: 4
    πŸ’ͺ 3
    :youtube: 5
    c
    • 2
    • 1
Powered by Linen
Title
a

Andreas

08/01/2022, 2:35 PM
Code Contest This is a simple flow that downloads videos of a playlist in parallel using pytube
#!/usr/bin/env python
import argparse
from pytube import Playlist, YouTube
from prefect import task, flow
from pydantic import HttpUrl
from prefect_dask import DaskTaskRunner

@task(retries=3, retry_delay_seconds=4)
def download_video(video: YouTube):
    video.streams.get_highest_resolution().download()
    return

@flow(task_runner=DaskTaskRunner)
def playlist_downloader(playlist_url: HttpUrl):
    p = Playlist(playlist_url)
    for video in p.videos:
        download_video.submit(video)
    return


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description='Downloads all youtubes videos in a playlist in parallel using the highest available resolution')
    parser.add_argument('playlist', help='a playlist url to download videos')
    
    args = parser.parse_args()
    playlist_downloader(args.playlist)
It is really great that using Prefect we are able to orchestrate this functionality in so few lines of code and use features like running stuff in parallel using Dask and having retry functionality πŸ˜›refect-duck:
:prefect2: 5
:marvin: 4
πŸ™Œ 9
:party-parrot: 4
πŸ’ͺ 3
:youtube: 5
c

Chris Reuter

08/01/2022, 2:38 PM
First submission! Very cool, thank you for sharing.
❀️ 1
View count: 3