https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
show-us-what-you-got
  • How to pass the AWS Certified ML - Speciality exam
    j

    Jeff Hale

    12/07/2022, 7:51 PM
    Have you considered taking the AWS Certified Machine Learning - Specialty exam? I took it a recently and couldn’t find many updated resources for it, so I wrote this post to help folks decide whether to take it and how to study for it.
    :aws: 5
    🙌 2
    :gratitude-thank-you: 1
    :thank-you: 1
    🔥 1
    👏🏽 1
  • a

    Anna Geller

    12/07/2022, 9:35 PM
    cross-posting a new community post on getting started - thanks for sharing @anurag ambuja 🙌
    🙌 7
    :party-parrot: 4
    🙏 3
  • k

    Khuyen Tran

    12/09/2022, 6:01 PM
    Hex is a modern data workspace that makes it easy to connect to data and create reproducible notebooks. This series of video will show you how to integrate notebooks into a Prefect pipeline with the prefect-hex collection. •

    Part 1: Benefits of Hex▾

    •

    Part 2: Integrate Hex with Prefect▾

    View the full Prefect playlist.
    🙌 12
    :marvin: 7
    :youtube: 6
    :upvote: 6
    🚀 3
    💙 2
    :prefect: 2
    :gratitude-thank-you: 2
    👏 3
    :party-parrot: 2
  • a

    Anna Geller

    12/21/2022, 7:13 PM
    How to get started with Prefect & GCP - agent deployed to a Compute Engine VM and flow runs to serverless Cloud Run containers - repo template with GitHub Actions automations included ✨
    ✨ 5
    💯 12
    :gcp: 10
    🙌 7
    j
    t
    • 3
    • 2
  • m

    Madison Schott

    12/29/2022, 4:51 PM
    Hey everyone! If you’re looking to get into analytics engineering or sharpen your skills/best practices in 2023, I just launched my first ebook- The ABCs of Analytics Engineering. This is a guide to all of the major concepts and tools you need to know to be successful. There’s a chapter on data quality, orchestration, data pipelines, documentation, modularity, freshness, testing, and more! https://madisonmae.gumroad.com/l/learnanalyticsengineering
    🔥 10
    🙌 3
    :blob-attention-gif: 2
    🎉 3
    :party-parrot: 2
    👀 4
    a
    • 2
    • 2
  • j

    Jeff Hale

    01/03/2023, 4:21 PM
    Happy new year, all! 🎉 If you’ve been around for a minute you know that at Prefect we add new features fast. Blink and you miss one. I made a recap of my ten favorite new enhancements since v2 GA five months ago. Check it out to see the highlights and let us know if you have other favorites. 🙂
    🎉 6
    💯 4
    :prefect: 4
    a
    • 2
    • 2
  • k

    Khuyen Tran

    01/03/2023, 4:26 PM
    This article and

    video▾

    will teach you how to build a full-stack ML application with Pydantic and Prefect.
    🙌 6
    🚀 5
    💯 5
    j
    • 2
    • 2
  • a

    Anna Geller

    01/05/2023, 4:26 PM
    cross-posting: new Sentry block from the community ✨
    🙌 8
    ⭐ 5
    :party-parrot: 4
    j
    • 2
    • 1
  • a

    Anna Geller

    01/09/2023, 1:55 PM
    Kudos to @Danilo Drobac for contributing this very detailed Prefect tutorial :gratitude-thank-you:
    🎉 5
    s
    d
    a
    • 4
    • 4
  • d

    Danilo Drobac

    01/09/2023, 1:58 PM
    Thanks @Anna Geller, and @Jeff Hale for the proof reading and support on finishing it! I enjoyed writing this one, hopefully it helps somebody at least 🙂 The following part about deployment will be up in the next 10 days (planned).
    💯 6
    🙌 5
    :gratitude-thank-you: 5
    🚀 6
    :party-parrot: 3
  • b

    Bryan Whiting

    01/09/2023, 8:36 PM
    I love the logger. How do I print the task’s inputs into the logger? Like: “Submitted task run ‘function(x=“Hello world”)’. I want to easily know which input it failed on
    k
    • 2
    • 2
  • s

    Stephen

    01/11/2023, 4:14 AM
    Giving writing a go.. here's one I made earlier regarding migrating flows from v1 to v2
    👍 7
    ❤️ 5
    :gratitude-thank-you: 3
    ⭐ 2
    a
    • 2
    • 1
  • a

    Anna Geller

    01/11/2023, 4:47 PM
    New post on how to use Prefect with Fivetran 😛refect:
    :party-parrot: 2
    :fivetran: 4
    👍 1
    5️⃣ 3
    :cloud2: 4
    :prefect: 4
    s
    • 2
    • 2
  • h

    Henning Holgersen

    01/13/2023, 6:55 PM
    I had some fun with Prefect 2, Azure Container Instances and ARM templates this week. If anyone wants to try it out, I have a small writeup and a quickstart to create an agent on ACI, a key vault (complete with secrets), access to it from ACI, the necessary prefect blocks, and a flow to demonstrate. https://dev.to/radbrt/prefect-in-azure-container-instances-2jph https://github.com/radbrt/prefect_aci
    👍 5
    ❤️ 6
    🎉 3
    :gratitude-thank-you: 2
    👏 1
    💯 1
    🙌 2
    r
    a
    c
    • 4
    • 4
  • r

    Ryan Peden

    01/14/2023, 9:10 PM
    Continuing the Azure theme, here's a quick guide on how to send Prefect logs to Azure Application Insights: https://discourse.prefect.io/t/how-to-send-prefect-logs-to-azure-application-insights/2202
    :gratitude-thank-you: 3
    👍 4
    :prefect-duck: 2
    🎉 2
  • k

    Khuyen Tran

    01/16/2023, 2:18 AM
    Do you want to use Prefect in your data science project, but don’t know how to get started? My article How to Structure an ML Project for Reproducibility and Maintainability will show you to use a data science template to create an ideal project structure.
    :gratitude-thank-you: 2
    👍 4
    🙌 2
    💯 3
    :prefect: 2
    🚀 3
  • d

    Danilo Drobac

    01/16/2023, 9:21 AM
    Hey folks - New Article on a Full Deployment of Prefect 2.0 with Prefect Cloud & GCP!
    🙌 5
    👍 3
    :gratitude-thank-you: 2
    💯 4
    :cloud2: 4
    🙌🏽 1
    a
    • 2
    • 1
  • c

    Chris Reuter

    01/17/2023, 7:25 PM
    Reposting for visibility Anyone else on Cloud 2 get a Prefect Wrapped?
    :gratitude-thank-you: 4
    :prefect: 4
    :typingcat: 9
    👍 4
    r
    • 2
    • 5
  • n

    Nico Neumann

    01/21/2023, 2:43 AM
    One year ago I opened the issue for arm64 support. Many thanks to ddelange and @Michael Adkins who made this possible with the latest prefect 2.7.9 release! I got the arm64 build running on my Raspberry Pi 4 (4GB RAM, 64Bit OS) using Prefect Cloud and running the prefect-agent in Portainer. @Anna Geller provided a great Dataflow Repository and I used the health check flow as the first flow on my pi. I am also looking forward to switch to AWS Graviton instances. Thanks to the whole prefect team for the amazing work! It is incredible to see how prefect evolved from 0.X ➡️ 1.X ➡️ 2.X. The Slack support as well as the stability and features of prefect are really awesome!
    ❤️ 10
    :blob-attention-gif: 7
    :gratitude-thank-you: 5
    a
    • 2
    • 3
  • j

    Jack

    01/23/2023, 5:04 AM
    Hello Prefect community! Over the weekend, I dove into the Prefect-GCP library and decided to build a data pipeline with Prefect, GCP, and dbt. This was a great learning experience and I wanted to share my learnings, check out a quick walkthrough here - https://jackskylord.medium.com/sky-pipe-prefect-gcp-dbt-data-pipeline-80561107c38f
    😍 9
    🐶 6
    :party-parrot: 3
    :gcp: 4
    🙌 7
    :gratitude-thank-you: 6
    :dbt: 4
    😛anda-dancing: 1
    a
    • 2
    • 2
  • a

    Anna Geller

    01/24/2023, 3:41 PM
    Many of you requested a diagram of Prefect concepts - we now have it! Big shoutout to @Taylor Curran for creating it 💙
    prefect_concept_map.pdf
    🥳 13
    :highfive: 10
    :thank-you: 11
    😛anda-dancing: 11
    :blob-attention-gif: 7
    :party-parrot: 5
    ❤️ 11
    🧑‍🎨🏽 1
    😒quirtle_cool: 4
    🧑‍🎨 18
    n
    j
    m
    • 4
    • 3
  • j

    Jeff Hale

    01/24/2023, 7:29 PM
    My PyData NYC talk

    Supercharge your Python Code with Blocks▾

    was just posted. Start at 9:38 if you want to hear how Prefect relates to Charles Babbage. 🙂
    👀 5
    :party-parrot: 6
    :prefect: 3
    🚀 2
    :cloud2: 3
    :prefect-duck: 2
    🧱 7
    :oh-yeah: 8
    :gratitude-thank-you: 3
    👍 4
    🙌 3
    ❤️ 4
    💯 2
    :blob-attention-gif: 1
    j
    • 2
    • 1
  • a

    Anna Geller

    01/26/2023, 4:32 PM
    Cross-posting Airbyte announcement for visibility
    🙌 1
  • a

    Anna Geller

    01/26/2023, 11:41 PM
    New post by @Andrew Huang on how to leverage the new prefect-openai collection to explain exception messages in plain English. Very cool use case, I highly recommend checking it out!
    ❤️ 9
    :marvin: 6
    :blob-attention-gif: 5
    :prefect-duck: 6
    :party-parrot: 5
  • a

    Anna Geller

    01/31/2023, 10:35 PM
    New post by @Nate about ELT using Airbyte, dbt, Snowflake, and of course... 😛refect:
    :marvin: 8
    :party-parrot: 9
    :wizard2: 6
    :blob-attention-gif: 6
    🧑‍🎨 2
    😛anda-dancing: 2
    t
    • 2
    • 1
  • b

    Bernard Greyling

    02/09/2023, 12:10 PM
    Hi all ! I've been playing around with the idea using prefect in conjunction with JuiceFS for a distributed runtime environment. In essence the idea is to use JuiceFS to mount a distributed files system to K8s pods to act as a shared virtual filesystem. To simplify local development one simply has to manage a the runtime environment variables appropriately. In this example repo I attempt to illustrate a basic local configuration. Any ideas and comments would be much appreciated! https://github.com/CrispyCrafter/JuicyPrefect
    🙌 3
    👍 3
    🧃 3
    :gratitude-thank-you: 2
    👏 7
    ⭐ 6
    a
    • 2
    • 1
  • a

    Amine Dirhoussi

    02/10/2023, 2:22 PM
    Hello Prefect Community, I started working on a distributed task queue library a few months back. The library is available as a python package to install a start using : daskqueue - pypi package For all its greatness, Dask implements a central scheduler (basically a simple tornado event loop) involved in every decision, which can sometimes create a central bottleneck. This is a pretty serious limitation when trying to use Dask in high-throughput situations. Daskqueue is a small python library built on top of Dask and Dask Distributed that implements a very lightweight Distributed Task Queue. Daskqueue also implements persistent queues for holding tasks on disk and surviving Dask cluster restart. This could be used as a special Executor to enqueue work. I have experiment the limits of the
    .map()
    method where mapping on a large Iterable will bring the Dask cluster to a halt. I also wrote an article about implementation details: https://medium.com/@aminedirhoussi1/daskqueue-dask-based-distributed-task-queue-6fb95517dfea Hope you enjoy it, can't wait to hear about your feedback 😉 !
    :gratitude-thank-you: 5
    🎉 4
    👀 5
    👏 6
    :marvin-duck: 5
    🙌 6
    👍 4
    a
    • 2
    • 1
  • s

    Serina

    02/17/2023, 4:50 PM
    set the channel description: A place for sharing knowledge about Prefect, tips and tricks about working with data, and best practices about workflow orchestration and working with Python Prefect Recipes: https://github.com/PrefectHQ/prefect-recipes
    🎉 4
    :party-parrot: 3
  • e

    Elisa Cerquozzi

    02/17/2023, 8:51 PM
    Looking forward to the newest Prefect Certified grads showing us what they got! Thank you all for a great session. :catjam:
    :prefect-duck: 7
    :party-parrot: 12
    :prefect: 4
  • j

    john enquist

    02/21/2023, 2:25 PM
    Hello Prefect Community! I know this may seem like small potatoes in the grand scheme of things but I wrote step by step documentation on how to install an instance of prefect 2 locally using .venv. really looking forward to getting a zip up sweatshirt!
    Setting Up Prefect 2 Locally using .venv.pdf
    😒nap-point: 3
    😆 4
    :party-parrot: 4
    🙌 8
    :thank-you: 2
    c
    a
    • 3
    • 6
Powered by Linen
Title
j

john enquist

02/21/2023, 2:25 PM
Hello Prefect Community! I know this may seem like small potatoes in the grand scheme of things but I wrote step by step documentation on how to install an instance of prefect 2 locally using .venv. really looking forward to getting a zip up sweatshirt!
Setting Up Prefect 2 Locally using .venv.pdf
😒nap-point: 3
😆 4
:party-parrot: 4
🙌 8
:thank-you: 2
c

Chris Reuter

02/21/2023, 3:15 PM
@john enquist DM me your email address! thanks for posting
a

Anna Geller

02/21/2023, 7:02 PM
haha marvins are not enough, nice one! thx for contributing!
@john enquist do you happen to have that as markdown or google doc? I could then share on Discourse or you could share on Discourse as well: discourse.prefect.io
🎉 1
👍 1
you can add it here https://discourse.prefect.io/t/setting-up-prefect-2-locally-using-venv-by-john-enquist/2428
j

john enquist

02/23/2023, 8:39 PM
Hi Anna! attached is a microsoft word version of the documentation, sorry for the late response!
SETTING UP PREFECT 2 LOCALLY USING venv.docx
:gratitude-thank-you: 1
a

Anna Geller

02/24/2023, 1:00 AM
awesome, added to the same Discourse topic 🚀
View count: 19