In case you missed PrefectLive with @Alix Cook today, we had a great chat about integrating Prefect in various organizational environments! If anyone missed the stream, you can catch it on . Thanks for coming on the show, Alix! 🎉

Hello all! I recently wrote a quick introduction on Prefect on some of the concepts of it after starting to utilize it at work, here it is if anyone wants a quick read! : • https://medium.com/@jackskylord/prefect-the-future-of-orchestration-93f4d2377885 Thanks 😛refect: !!

For those interested on trying Prefect 2, here's a quick repository a did to follow the prefect tutorial more easily on your computer. Simply do a

git clone <https://github.com/jmprovencher/prefect-tutorial>

and follow the steps in

README.md

and

TUTORIAL.md

There are lots of ways to use Kubernetes with Prefect. I’ll be exploring a number of them in a series of posts. Here’s the first post, a step-by-step introduction to running flows locally, in K8s, with S3 storage, and Prefect Cloud. 😛refect::kubernetes:

Many logging, APM, and DevOps tools ingest and aggregate logs. Most of them, however, only work with file-based logs. Prefect doesn't write to log files out of the box, but it's easy to set it up if you need it. In this Discourse post, I walk through the required steps: https://discourse.prefect.io/t/how-to-stream-prefect-logs-to-a-file/1963

To all users who are still on Prefect 1, this post dives into reasons why migrating to Prefect 2 is worth it 🙌

Hex is a modern data workspace that makes it easy to connect to data and create reproducible notebooks. This article will show you how to integrate notebooks into a Prefect pipeline with the prefect-hex collection.

Hi everyone, Nick here from Fivetran:fivetran:. I've been building a Fivetran Prefect Collection with the Prefect Integrations team, and we just announced it yesterday! Thank you to everyone at Prefect that's been building it with me and for everyone who has tried it so far! • Read the announcement • 4 Questions on the Fivetran Prefect Collection with @Emerson Franks • A Fivetran Prefect Collection Walkthrough

Looking to use Prefect with K8s? I just published a second post in the series that shows three ways you can customize Kubernetes Job blocks for your deployments. Check it out if you’re into that kind of thing. 🙂

Check out this article I wrote on using Fivetran with Prefect 2.0! It includes tips for migrating from the Fivetran tasks in 1.0 to those in 2.0 🙂 https://madison-schott.medium.com/prefect-2-0-fivetran-a-perfect-pair-6589568f7380

I wrote my hacked together version of prefect 1's

wait_for_flow_run

so that I can run sub flows concurrently without introducing all the async stuff I was trying to avoid. I ended up using the rest api. Would there have been an easier way to go about this? seems to do the job ✅

import time

import requests
from prefect import get_run_logger, task
from prefect.deployments import run_deployment

from aws_helpers import Ssm

@task
def wait_for_deployment(flow_run_id: str, log_polling_interval: int = 25, max_run_time: int = 1000):

    logger = get_run_logger()

    api_key = Ssm.get_secret("/prefect/api_key")
    api_url = Ssm.get_secret("/prefect/api_url")

    while True:

        flow_run = request_prefect_api(path=f"/flow_runs/{flow_run_id}", api_key=api_key, api_url=api_url)
        <http://logger.info|logger.info>(f"{flow_run['name']} : {flow_run['state_type']}")

        state = flow_run["state_type"]

        if state == "COMPLETED":
            break
        if state == "FAILED":
            raise Exception(f"Deployment: {flow_run['name']}, Flow run failed: {flow_run['state']['message']}")
        if flow_run["total_run_time"] > max_run_time:
            raise Exception(f"Flow run exceeded max run time of {max_run_time} seconds: marking as failed")

        time.sleep(log_polling_interval)


def request_prefect_api(path: str, api_key: str, api_url: str):
    url = f"{api_url}{path}"
    headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
    return requests.get(url, headers=headers).json()

@Maikel Penz - thank you for your write-up about how you use Prefect along with Databricks and AWS at Plexure. 🙌 https://medium.com/task-group/reshaping-data-engineering-at-plexure-5897bf398b2b

It takes adding just two lines to a Docker image to bake your Prefect flow code in for Kubernetes Job. Then you don’t need a storage block at all! See my new blog post for an example. 🚀

Have you considered taking the AWS Certified Machine Learning - Specialty exam? I took it a recently and couldn’t find many updated resources for it, so I wrote this post to help folks decide whether to take it and how to study for it.

cross-posting a new community post on getting started - thanks for sharing @anurag ambuja 🙌

Hex is a modern data workspace that makes it easy to connect to data and create reproducible notebooks. This series of video will show you how to integrate notebooks into a Prefect pipeline with the prefect-hex collection. • • View the full Prefect playlist.

How to get started with Prefect & GCP - agent deployed to a Compute Engine VM and flow runs to serverless Cloud Run containers - repo template with GitHub Actions automations included ✨

Hey everyone! If you’re looking to get into analytics engineering or sharpen your skills/best practices in 2023, I just launched my first ebook- The ABCs of Analytics Engineering. This is a guide to all of the major concepts and tools you need to know to be successful. There’s a chapter on data quality, orchestration, data pipelines, documentation, modularity, freshness, testing, and more! https://madisonmae.gumroad.com/l/learnanalyticsengineering

Happy new year, all! 🎉 If you’ve been around for a minute you know that at Prefect we add new features fast. Blink and you miss one. I made a recap of my ten favorite new enhancements since v2 GA five months ago. Check it out to see the highlights and let us know if you have other favorites. 🙂

This article and will teach you how to build a full-stack ML application with Pydantic and Prefect.

cross-posting: new Sentry block from the community ✨

Kudos to @Danilo Drobac for contributing this very detailed Prefect tutorial :gratitude-thank-you:

Thanks @Anna Geller, and @Jeff Hale for the proof reading and support on finishing it! I enjoyed writing this one, hopefully it helps somebody at least 🙂 The following part about deployment will be up in the next 10 days (planned).

I love the logger. How do I print the task’s inputs into the logger? Like: “Submitted task run ‘function(x=“Hello world”)’. I want to easily know which input it failed on

Giving writing a go.. here's one I made earlier regarding migrating flows from v1 to v2

New post on how to use Prefect with Fivetran 😛refect:

I had some fun with Prefect 2, Azure Container Instances and ARM templates this week. If anyone wants to try it out, I have a small writeup and a quickstart to create an agent on ACI, a key vault (complete with secrets), access to it from ACI, the necessary prefect blocks, and a flow to demonstrate. https://dev.to/radbrt/prefect-in-azure-container-instances-2jph https://github.com/radbrt/prefect_aci

Continuing the Azure theme, here's a quick guide on how to send Prefect logs to Azure Application Insights: https://discourse.prefect.io/t/how-to-send-prefect-logs-to-azure-application-insights/2202

Do you want to use Prefect in your data science project, but don’t know how to get started? My article How to Structure an ML Project for Reproducibility and Maintainability will show you to use a data science template to create an ideal project structure.