https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
show-us-what-you-got
  • j

    Jeff Hale

    11/16/2022, 10:24 PM
    In case you missed PrefectLive with @Alix Cook today, we had a great chat about integrating Prefect in various organizational environments! If anyone missed the stream, you can catch it on

    YouTube▾

    . Thanks for coming on the show, Alix! 🎉
    🙌 4
    :gratitude-thank-you: 4
    :hattip: 2
  • j

    Jack

    11/17/2022, 1:11 PM
    Hello all! I recently wrote a quick introduction on Prefect on some of the concepts of it after starting to utilize it at work, here it is if anyone wants a quick read! : • https://medium.com/@jackskylord/prefect-the-future-of-orchestration-93f4d2377885 Thanks 😛refect: !!
    :gratitude-thank-you: 7
    🐶 7
    :prefect: 6
    🙌 8
    j
    1 reply · 2 participants
  • j

    Jean-Michel Provencher

    11/17/2022, 9:27 PM
    For those interested on trying Prefect 2, here's a quick repository a did to follow the prefect tutorial more easily on your computer. Simply do a
    git clone <https://github.com/jmprovencher/prefect-tutorial>
    and follow the steps in
    README.md
    and
    TUTORIAL.md
    💯 8
    ❤️ 7
    🔥 7
    :gratitude-thank-you: 5
    🙌 5
    j
    1 reply · 2 participants
  • j

    Jeff Hale

    11/21/2022, 5:35 PM
    There are lots of ways to use Kubernetes with Prefect. I’ll be exploring a number of them in a series of posts. Here’s the first post, a step-by-step introduction to running flows locally, in K8s, with S3 storage, and Prefect Cloud. 😛refect::kubernetes:
    ✅ 1
    :kubernetes: 6
    🙌 6
    🚀 4
    :gratitude-thank-you: 4
  • r

    Ryan Peden

    11/23/2022, 4:59 PM
    Many logging, APM, and DevOps tools ingest and aggregate logs. Most of them, however, only work with file-based logs. Prefect doesn't write to log files out of the box, but it's easy to set it up if you need it. In this Discourse post, I walk through the required steps: https://discourse.prefect.io/t/how-to-stream-prefect-logs-to-a-file/1963
    👀 6
    🙌 4
    :gratitude-thank-you: 5
    ❤️ 4
    💯 4
    🚀 6
  • a

    Anna Geller

    11/28/2022, 5:38 PM
    To all users who are still on Prefect 1, this post dives into reasons why migrating to Prefect 2 is worth it 🙌
    ❤️ 11
    :long_drum: 5
    🔥 7
    :gratitude-thank-you: 3
  • k

    Khuyen Tran

    11/29/2022, 4:19 PM
    Hex is a modern data workspace that makes it easy to connect to data and create reproducible notebooks. This article will show you how to integrate notebooks into a Prefect pipeline with the prefect-hex collection.
    🙌 7
    🔥 6
    :marvin: 5
    📓 1
    👍 1
    o
    2 replies · 2 participants
  • n

    Nick Acosta

    11/29/2022, 11:18 PM
    Hi everyone, Nick here from Fivetran:fivetran:. I've been building a Fivetran Prefect Collection with the Prefect Integrations team, and we just announced it yesterday! Thank you to everyone at Prefect that's been building it with me and for everyone who has tried it so far! • Read the announcement • 4 Questions on the Fivetran Prefect Collection with @Emerson Franks • A Fivetran Prefect Collection Walkthrough
    🙌 12
    ❤️ 10
    :fivetran: 10
    a
    2 replies · 2 participants
  • j

    Jeff Hale

    11/30/2022, 2:45 PM
    Looking to use Prefect with K8s? I just published a second post in the series that shows three ways you can customize Kubernetes Job blocks for your deployments. Check it out if you’re into that kind of thing. 🙂
    :kubernetes: 2
    💙 1
    🙌 4
    :gratitude-thank-you: 5
    :prefect: 2
  • m

    Madison Schott

    11/30/2022, 5:55 PM
    Check out this article I wrote on using Fivetran with Prefect 2.0! It includes tips for migrating from the Fivetran tasks in 1.0 to those in 2.0 🙂 https://madison-schott.medium.com/prefect-2-0-fivetran-a-perfect-pair-6589568f7380
    🙌 11
    :fivetran: 8
    :prefect-duck: 5
    :party-parrot: 2
    🍪 4
    a
    1 reply · 2 participants
  • b

    Ben Muller

    12/05/2022, 12:19 AM
    I wrote my hacked together version of prefect 1's
    wait_for_flow_run
    so that I can run sub flows concurrently without introducing all the async stuff I was trying to avoid. I ended up using the rest api. Would there have been an easier way to go about this? seems to do the job ✅
    import time
    
    import requests
    from prefect import get_run_logger, task
    from prefect.deployments import run_deployment
    
    from aws_helpers import Ssm
    
    @task
    def wait_for_deployment(flow_run_id: str, log_polling_interval: int = 25, max_run_time: int = 1000):
    
        logger = get_run_logger()
    
        api_key = Ssm.get_secret("/prefect/api_key")
        api_url = Ssm.get_secret("/prefect/api_url")
    
        while True:
    
            flow_run = request_prefect_api(path=f"/flow_runs/{flow_run_id}", api_key=api_key, api_url=api_url)
            <http://logger.info|logger.info>(f"{flow_run['name']} : {flow_run['state_type']}")
    
            state = flow_run["state_type"]
    
            if state == "COMPLETED":
                break
            if state == "FAILED":
                raise Exception(f"Deployment: {flow_run['name']}, Flow run failed: {flow_run['state']['message']}")
            if flow_run["total_run_time"] > max_run_time:
                raise Exception(f"Flow run exceeded max run time of {max_run_time} seconds: marking as failed")
    
            time.sleep(log_polling_interval)
    
    
    def request_prefect_api(path: str, api_key: str, api_url: str):
        url = f"{api_url}{path}"
        headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
        return requests.get(url, headers=headers).json()
    a
    y
    19 replies · 3 participants
  • c

    Chris Reuter

    12/06/2022, 3:22 PM
    @Maikel Penz - thank you for your write-up about how you use Prefect along with Databricks and AWS at Plexure. 🙌 https://medium.com/task-group/reshaping-data-engineering-at-plexure-5897bf398b2b
    🔥 10
    🙌 5
    ❤️ 4
    💯 4
    :gratitude-thank-you: 4
    a
    m
    2 replies · 3 participants
  • Run your flow as a Docker image with Kubernetes
    j

    Jeff Hale

    12/07/2022, 7:18 PM
    It takes adding just two lines to a Docker image to bake your Prefect flow code in for Kubernetes Job. Then you don’t need a storage block at all! See my new blog post for an example. 🚀
    :prefect: 1
    👍 3
    :marvin-duck: 1
    🚀 1
    🙌 1
    :gratitude-thank-you: 1
  • How to pass the AWS Certified ML - Speciality exam
    j

    Jeff Hale

    12/07/2022, 7:51 PM
    Have you considered taking the AWS Certified Machine Learning - Specialty exam? I took it a recently and couldn’t find many updated resources for it, so I wrote this post to help folks decide whether to take it and how to study for it.
    :aws: 5
    🙌 2
    :gratitude-thank-you: 1
    :thank-you: 1
    🔥 1
    👏🏽 1
  • a

    Anna Geller

    12/07/2022, 9:35 PM
    cross-posting a new community post on getting started - thanks for sharing @anurag ambuja 🙌
    🙌 7
    :party-parrot: 4
    🙏 3
  • k

    Khuyen Tran

    12/09/2022, 6:01 PM
    Hex is a modern data workspace that makes it easy to connect to data and create reproducible notebooks. This series of video will show you how to integrate notebooks into a Prefect pipeline with the prefect-hex collection. •

    Part 1: Benefits of Hex▾

    •

    Part 2: Integrate Hex with Prefect▾

    View the full Prefect playlist.
    🙌 12
    :marvin: 7
    :youtube: 6
    :upvote: 6
    🚀 3
    💙 2
    :prefect: 2
    :gratitude-thank-you: 2
    👏 3
    :party-parrot: 2
  • a

    Anna Geller

    12/21/2022, 7:13 PM
    How to get started with Prefect &amp; GCP - agent deployed to a Compute Engine VM and flow runs to serverless Cloud Run containers - repo template with GitHub Actions automations included ✨
    ✨ 5
    💯 12
    :gcp: 10
    🙌 7
    j
    t
    2 replies · 3 participants
  • m

    Madison Schott

    12/29/2022, 4:51 PM
    Hey everyone! If you’re looking to get into analytics engineering or sharpen your skills/best practices in 2023, I just launched my first ebook- The ABCs of Analytics Engineering. This is a guide to all of the major concepts and tools you need to know to be successful. There’s a chapter on data quality, orchestration, data pipelines, documentation, modularity, freshness, testing, and more! https://madisonmae.gumroad.com/l/learnanalyticsengineering
    🔥 10
    🙌 3
    :blob-attention-gif: 2
    🎉 3
    :party-parrot: 2
    👀 4
    a
    2 replies · 2 participants
  • j

    Jeff Hale

    01/03/2023, 4:21 PM
    Happy new year, all! 🎉 If you’ve been around for a minute you know that at Prefect we add new features fast. Blink and you miss one. I made a recap of my ten favorite new enhancements since v2 GA five months ago. Check it out to see the highlights and let us know if you have other favorites. 🙂
    🎉 6
    💯 4
    :prefect: 4
    a
    2 replies · 2 participants
  • k

    Khuyen Tran

    01/03/2023, 4:26 PM
    This article and

    video▾

    will teach you how to build a full-stack ML application with Pydantic and Prefect.
    🙌 6
    🚀 5
    💯 5
    j
    2 replies · 2 participants
  • a

    Anna Geller

    01/05/2023, 4:26 PM
    cross-posting: new Sentry block from the community ✨
    🙌 8
    ⭐ 5
    :party-parrot: 4
    j
    1 reply · 2 participants
  • a

    Anna Geller

    01/09/2023, 1:55 PM
    Kudos to @Danilo Drobac for contributing this very detailed Prefect tutorial :gratitude-thank-you:
    🎉 5
    s
    d
    +1
    4 replies · 4 participants
  • d

    Danilo Drobac

    01/09/2023, 1:58 PM
    Thanks @Anna Geller, and @Jeff Hale for the proof reading and support on finishing it! I enjoyed writing this one, hopefully it helps somebody at least 🙂 The following part about deployment will be up in the next 10 days (planned).
    💯 6
    🙌 5
    :gratitude-thank-you: 5
    🚀 6
    :party-parrot: 3
  • b

    Bryan Whiting

    01/09/2023, 8:36 PM
    I love the logger. How do I print the task’s inputs into the logger? Like: “Submitted task run ‘function(x=“Hello world”)’. I want to easily know which input it failed on
    k
    2 replies · 2 participants
  • s

    Stephen

    01/11/2023, 4:14 AM
    Giving writing a go.. here's one I made earlier regarding migrating flows from v1 to v2
    👍 7
    ❤️ 5
    :gratitude-thank-you: 3
    ⭐ 2
    a
    1 reply · 2 participants
  • a

    Anna Geller

    01/11/2023, 4:47 PM
    New post on how to use Prefect with Fivetran 😛refect:
    :party-parrot: 2
    :fivetran: 4
    👍 1
    5️⃣ 3
    :cloud2: 4
    :prefect: 4
    s
    2 replies · 2 participants
  • h

    Henning Holgersen

    01/13/2023, 6:55 PM
    I had some fun with Prefect 2, Azure Container Instances and ARM templates this week. If anyone wants to try it out, I have a small writeup and a quickstart to create an agent on ACI, a key vault (complete with secrets), access to it from ACI, the necessary prefect blocks, and a flow to demonstrate. https://dev.to/radbrt/prefect-in-azure-container-instances-2jph https://github.com/radbrt/prefect_aci
    👍 5
    ❤️ 6
    🎉 3
    :gratitude-thank-you: 2
    👏 1
    💯 1
    🙌 2
    r
    a
    +1
    4 replies · 4 participants
  • r

    Ryan Peden

    01/14/2023, 9:10 PM
    Continuing the Azure theme, here's a quick guide on how to send Prefect logs to Azure Application Insights: https://discourse.prefect.io/t/how-to-send-prefect-logs-to-azure-application-insights/2202
    :gratitude-thank-you: 3
    👍 4
    :prefect-duck: 2
    🎉 2
  • k

    Khuyen Tran

    01/16/2023, 2:18 AM
    Do you want to use Prefect in your data science project, but don’t know how to get started? My article How to Structure an ML Project for Reproducibility and Maintainability will show you to use a data science template to create an ideal project structure.
    :gratitude-thank-you: 2
    👍 4
    🙌 2
    💯 3
    :prefect: 2
    🚀 3
  • d

    Danilo Drobac

    01/16/2023, 9:21 AM
    Hey folks - New Article on a Full Deployment of Prefect 2.0 with Prefect Cloud &amp; GCP!
    🙌 5
    👍 3
    :gratitude-thank-you: 2
    💯 4
    :cloud2: 4
    🙌🏽 1
    a
    1 reply · 2 participants
Powered by Linen
Title
d

Danilo Drobac

01/16/2023, 9:21 AM
Hey folks - New Article on a Full Deployment of Prefect 2.0 with Prefect Cloud &amp; GCP!
🙌 5
👍 3
:gratitude-thank-you: 2
💯 4
:cloud2: 4
🙌🏽 1
a

Anna Geller

01/16/2023, 12:04 PM
Thanks so much, very detailed and thorough, can't wait to see the next part with Terraform as well 🚀
👍 2
View count: 3