I genuinely don't know how @Jeremiah finds time for creating content next to being a CEO, a dad, a Twitter 🌠 and very actively contributing to the codebase and product, but here are two recent pieces of content about dataflow and the problem of orchestration created by Jeremiah: • Data Exchange - podcast • Negative Engineering and the Art of Failing Successfully - blog post This might be especially interesting to those who want to find out more about the problem Prefect tries to solve and the product philosophy behind it. And why you should never say "just".

Thank to @Nate for writing a cool (and hilarious) guide demonstrating how to turn your existing Python code into scheduled, repeatable, Prefect flows. Check it out! https://www.prefect.io/blog/your-python-job-is-already-a-prefect-flow-it-just-doesnt-know-it-yet/

Knowledge management sounds cool but it rarely works. Why? I recently had a great conversation with Stephanie Barnes the creator of Radical Knowledge Management. Stephanie advocates for curiosity, playfulness and creative learning as the cornerstones of the journey towards knowledge management, principles that (as she says) have been educated out of us through production-line thinking. Read / Listen👇🏻 https://www.discoveringdata.com/podcast/034-creative-learning

set the channel topic: Built something cool with Prefect? Wrote an article or a social media post featuring Prefect? Show us what you got!

set the channel description: A place for sharing knowledge about Prefect, tips and tricks about working with data, and best practices about workflow orchestration and working with Python

Hey folks 👋 Here you can find a blog post from the awesome @Stefano Cascavilla explaining how we do Data Engineering at Cloud Academy! https://cloudacademy.com/blog/how-do-we-transform-and-model-data-at-cloud-academy/

Also check out @Madison Schott’s post from yesterday about using P, dbt and S to monitor data freshness: https://towardsdatascience.com/how-fresh-are-your-data-sources-e8db53cf4653

A fresh quick start tutorial is out today! Check out Getting Started with Prefect 2.0 by @Peyton Runyan for a step-by-step tutorial for integrating Prefect into your code.Peyton's walkthrough is an agile method for adopting Prefect quickly with just a few lines of code and then expands to scheduling, work queues and agents. Want to write your own Prefect content? It's easy! If you have any questions, we're here to help. Just post right in this channel.

You may have noticed our recently updated resources section on our website, appropriately titled The Guide! It features community content, tutorials, guides, company news, videos, podcasts and more. Enjoy exploring, and we hope that you find it useful during your Prefect Journey. https://www.prefect.io/guide/ Also check out our clutch Zelda reference on our announcement tweet ❤️

This message was deleted.

I wrote about the trends in data orchestrations, and of course, prefect is a part of it 🙂 Hope it's helpful, and looking forward to your comments. I'm covering topics such as: • The shift from Data Pipelines to Data Products • Data-aware Pipelines that know about the inner life of a task • Why a declarative approach with higher-level abstractions helps • How abstractions improve reuse code between complex cloud environments • Where do we come from with the evolution of Data Pipeline Orchestration • Open-source data orchestration tools such as Airflow, Prefect, Dagster https://airbyte.com/blog/data-orchestration-trends

Have you ever wondered what negative engineering is? I created to clarify the difference between negative engineering and positive engineering.

I just posted about . This video meant to be a short and fun watch

I just posted in case you want to wrap your head around caching

I often use GitHub feed to get updated with cool and trending libraries. However, I don’t care about 70% of what I saw on my feed so I built a workflow to get the data from GitHub and create a dashboard. View my article on this project here.

I just posted a and how to use retries with Prefect

Why do some financial professionals like to "make it rain" 💦☔? I made a program to figure that out. My program generates spurious stock correlations between stock price and rainfall based on historical movers & shakers. Obviously not investment advice. Github: https://github.com/KevinFocke/noisy_stocks_data_orchestrator Website: https://www.noisystocks.com/

Do you know that you can add flows under another flow in Prefect 2.0? will explain the difference between task, flow, subflow, and how to create a subflow.

Dunno if useful for anyone but I wanted a quick view of the states of my flows so put together this CLI script

Some Prefect Orion examples running self-contained in a local kubernetes cluster. Flows • Dask flow using a dask local cluster • Dask kubes flow using an ephemeral dask cluster on Kubernetes • Parameterized flow using a custom Docker image containing additional modules • Ray flow that runs on an existing ray cluster • Sub flow that is trigger by a parent flow Deployments to Kubernetes using • the default OrionPackager to store the flow source in Orion • the default OrionPackager to store the flow import path in Orion • the FilePackager to store the flow source in S3 (minio) Batteries (mostly) included. 🔋

This message was deleted.

Did you know that you can now send Slack notifications based on the state of your flow by simply clicking on the UI in Prefect 2.0b8 and above? This article shows how you can do exactly that.

Many of you requested a real-time streaming recipe - it's finally released!

Hey, everyone! We constructed a simple Prefect-1.0 AWS ECS Boilerplate to deploy multiple flows with dependencies using GitHub Storage and ECSRun. We set up the ECS infrastructure with Terraform; with GitHub Actions we build and push a Docker image to ECR and register the flows to Prefect; and, finally, we run the flows on Prefect Cloud UI with a local ECS Agent. This boilerplate facilitates: - Import from flow modules in different directories - Automatic Prefect flow configuration and registration - The use of GitHub Secrets and dotenv variables - ECS Cluster setup with Terraform - The connection to ECR and Prefect using GitHub Actions Check out the repository here: https://github.com/eidos-ai/prefect-aws-boilerplate. We hope this template is useful to others, and we look forward to reading your feedback!

@George Coyne has recently published this great blog post to help you migrate your Prefect 1.0 flows to Prefect 2.0 cloud2

Hey all, I created a simple example showing how to deploy a Prefect agent to a docker on Google Cloud Compute Engine, using Github Actions. Flow stored in Google Cloud Storage. I had previously posted this when Prefect 2 was in beta, and just updated it for 2.0.1. https://github.com/BennyJW/prefect2-docker-gce

Hey everyone, I would like to share a simple pattern for single deployment file to execute multiple flows on Prefect Orion 2.0.1: https://www.notion.so/jlom/Prefect-Orion-Deployment-Guide-4cc9393bd4c943a38777a92027522cf8

For those who want to send notifications in 2.0, check out this article. I just updated the code so that it will work with the latest version of 2.0.

Loving all of the community contributed content! We'll add these to The Guide and would encourage anyone to post their templates/patterns/examples here.

Code Contest This is a simple flow that downloads videos of a playlist in parallel using pytube

#!/usr/bin/env python
import argparse
from pytube import Playlist, YouTube
from prefect import task, flow
from pydantic import HttpUrl
from prefect_dask import DaskTaskRunner

@task(retries=3, retry_delay_seconds=4)
def download_video(video: YouTube):
    video.streams.get_highest_resolution().download()
    return

@flow(task_runner=DaskTaskRunner)
def playlist_downloader(playlist_url: HttpUrl):
    p = Playlist(playlist_url)
    for video in p.videos:
        download_video.submit(video)
    return


if __name__ == "__main__":
    parser = argparse.ArgumentParser(description='Downloads all youtubes videos in a playlist in parallel using the highest available resolution')
    parser.add_argument('playlist', help='a playlist url to download videos')
    
    args = parser.parse_args()
    playlist_downloader(args.playlist)

It is really great that using Prefect we are able to orchestrate this functionality in so few lines of code and use features like running stuff in parallel using Dask and having retry functionality prefect duck