The latest versions of Prefect make it easy to use GitHub as the storage and Docker as the infrastructure using blocks. In my article Create Robust Data Pipelines with Prefect, Docker, and GitHub, you will learn how to use Prefect to easily store your flow in GitHub and execute it in a Docker container.

wrote up a little gist around how I was able to use blocks to allow for cleaning up and managing flow runs in a restrictive environment (and was able to solve the problem i described in this thread) https://gist.github.com/acookin/aa5f0d718997e7a645cbc160e2c94a16

Today we published: • 2.4.0 release blog post • @Oscar Björhn joined us on Prefect Live - you can

watch the recording▾

if you missed it! Thanks to Oscar for being such a great community member and for joining us today 💙

Checkout how Quilt.AI used Prefect to empower its scalable & flexible ML & Data platform. https://www.prefect.io/guide/case-studies/prefect-quilt-ai/

Posting here so that it doesn't get buried in community - Thank you @Hallen Maia for your excellent work!

Hi all! This is a really simple one. I’m brand new to the platform, just a few days in. I already get too many Slack notifications, so I prefer to keep app alerts/failures/exceptions in Google Chat. That way I know not to ignore it like my other slack notifications 😂 jk. It turns out the google chat JSON structure for simple text works exactly the same as the slack one, which means you can simply paste in the google chat webhook in place of the

slack webhook

for block/notifications in the prefect ui. If you haven’t used google chat webhooks, it’s even slightly easier than slack. https://developers.google.com/chat/how-tos/webhooks#create_a_webhook

If you want to read more about: • the difference between tasks, flows, and subflows • when to return data 📊, state ✅, or Prefect future 🔮 then add this post to your reading list for the week: https://medium.com/p/5eaabdfbb70e Have a great start to the new week!

New blog post by @justabill 🔥

If any one is interested in how you can: 1. Define Deployments in a separate repo from flows 2. Define Deployments that run any docker container and command (not just prefect flows) Here is an [experimental repo](https://github.com/datavaluepeople/prefect_deployment_experiment) that shows how. Thanks @Anna Geller and @Khuyen Tran for the support on this.

Prefect sold out, smh. But seriously, what are the chances of getting this flow run name?

Scheduled serverless flows in 5 minutes - this post shows how you can use alternative tools (e.g., GitHub Actions workflow or legacy orchestrators) to run Prefect flows on schedule

Can I get feedback/corrections on this sequence diagram? I’m using

prefect-aws

and

prefect-dask

+

dask-cloudprovider

for autoscaling

from prefect import flow, task from prefect import flow, task import yfinance as yf from time import sleep from random import randint from pandas import DataFrame from datetime import timedelta @task(retries = 5, retry_delay_seconds = 5) def getData(ticker): print('before: ', ticker) df = yf.download(ticker) print('after ') return df @task def displayTicker(df:DataFrame): print(df) @flow def getTickerData(symbols): print('in getTickerData') dfs = getData.map(symbols) print('After getTickerData') #displayTicker(dfs) return dfs if name == "__main__": print(' --------------------------------------- ') print(' --------------------------------------- ') ticker = ['DE','GE','IBM','CSCO'] stockTable = getTickerData(ticker) print ('*** ', ticker, ' * ') print (stockTable) print(' --------------------------------------- ') print(' --------------------------------------- ')

Shoutout to @Oscar Björhn for giving excellent product feedback that was directly used to build a new feature. The

run_deployment

utility was driven, a.o., by his feedback in the Community and -- the timestamp in the video shows the reason why Oscar implemented his own way of running flows from deployment, and the final implementation in Prefect 2.5 even used the same name as the one implemented by Oscar! 🚀 Great work, and really glad to have you in the community! 💙

How to build a data platform with Prefect P*+ dbt* dbt + Snowflake ❄️ - part 1 (conceptual post)

My article on Create a Maintainable Data Pipeline with Prefect and DVC Example project.

I have pondered deployment patterns in Prefect 2 a little more, and come up with something I might like: https://github.com/radbrt/orion_flows Hopefully the readme is understandable, but the gist of it is to create deployments in Python files (the

Deployment

class), in separate files that imports the flow and creates+applies a deployment. These files have a common naming pattern, all ending with

.deployment.py

, so that github action can pick them up and run them. This way, the flow file itself remains clean, and we don’t hardcode any config in the github action yaml, making it fully configurable. Thanks to @Emil Christensen for explaining the point of deployments to me in a way I finally understood. Oh, and there is also a cookiecutter template to create a new skeleton flow.

👋 I just released https://github.com/madkinsz/prefect-kv which is a key-value store for use in your flows/tasks built on top of Prefect’s JSON block!

Hey all, huge shout out to @Serina for revamping our Prefect Recipes repo! 🧑‍🍳 There's several recipes to help getting started deploying agents on different infrastructure such as k8s or docker containers. If you haven't checked it out yet, it's got lots of great examples and is continuing to grow. It's a great spot to contribute with big ideas 💡

I wrote a guide to help folks get started contributing to open source in Towards Data Science! If you haven’t tried before, it’s easier than you might think. At the end of the post you’ll see a sneak peak of Prefect’s #hactoberfest! 🎃

If you're looking for a good way to explain what Prefect is to your boss, co-worker, significant other, grandparent, neighbor or that person in the grocery - we made this for you:

As promised in Anna's release Prefect 2.6.4 release announcement, I just published a post introducing the new Azure Container Instances infrastructure block and showing how to set it up to run your flows: https://medium.com/the-prefect-blog/serverless-prefect-flows-with-azure-container-instances-f2442ebc9343

I’ve shared parts of the code that I have implemented to orchestrate an ingestion of spatial data datasets which need to be paused for approval https://discourse.prefect.io/t/implementing-waiting-for-approval-in-prefect-2/1558/6?u=francbartoli. I liked to apply the workaround suggested by @Anna Geller which is working nicely but also I’d love to have this feature handled natively in Orion. Very soon I will share the whole code publicly

If you want to send an alert when a flow fails by adding only one decorator, use prefect-alert.

from prefect import flow, task 
from prefect.blocks.notifications import SlackWebhook
from prefect_alert import alert_on_failure

@task
def may_fail():
    raise ValueError()

@alert_on_failure(block_type=SlackWebhook, block_name="test")
@flow
def failed_flow():
    res = may_fail()
    return res

if __name__=="__main__":
    failed_flow()

Thanks @Ryan Peden for contributing the collection!

Have you ever wanted to automatically rerun modified components of a data pipeline? In this article, you will learn how to: • Use GitHub Actions to run a workflow when you push a commit • Use DVC to run stages with modified dependencies Code used in the article.

If you use Prefect on Google Cloud, you might enjoy this post I wrote about how to run Prefect flows as Google Cloud Run jobs: https://medium.com/the-prefect-blog/serverless-prefect-flows-with-google-cloud-run-jobs-23edbf371175

Hey all - I seem to get unlimited help from all of the incredible prefect peeps on the #prefect-community channel, so thought that in some way/shape/form may be useful to others. It is very much specific to my use case, but maybe there are pieces that others can use. It is a CI pattern I am using for a repository of ECSTask flows that run across different workspaces in Prefect 2.0. The scripts might help out others. Have a look and feel free to shoot me any questions. https://github.com/bennnym/prefect-ci-pattern

This is similar to how we have organized things too, except we just kept a list in python over a yaml config. Thanks for sharing.

This is so simple, yet quite amazing:

from typing import Literal
from prefect import flow

AllowedStrings = Literal["A", "B"]

@flow
def example(str_input: AllowedStrings):
    print(str_input)

example("C")

--->

11:15:58.598 | INFO    | prefect.engine - Created flow run 'blond-auk' for flow 'example'
11:15:58.598 | ERROR   | Flow run 'blond-auk' - Validation of flow parameters failed with error: ParameterTypeError: Flow run received invalid parameters:
 - str_input: unexpected value; permitted: 'A', 'B'

cross-sharing from @Eden - post on medium kubernetes gcp