Code Contest What if we have many (and by many I mean MANY) subflows? In fact, let's just recursively create sub-flows while begging for a Prefect keyboard and/or a Marvin Rubber Duck 😆. See the first comment for code. As one might expect, Prefect runs into trouble when

subflow_count

(the number of subflows a flow generates aka breadth of the recursion tree) and

levels

(how many chained subflows there are aka the depth of the recursion tree) are large, 25 and 10 here respectively. That's 10^25 - 1 flows, truly absurd with long chains of subflows. When it fails, I get

OSError: [Errno 24] Too many open files

. I noticed that the crashed flows don't get marked as crashed but stay with the

running

status. I'm not sure there's an easy way for you to resolve that though. On the positive side, my computer didn't crash! At smaller values, e.g.

subflow_count=8

and

levels=3

, it runs just fine. My team and I are excited to use Prefect in the upcoming NASA PUNCH mission to prepare scientific data of the Sun. It's really helped make our pipeline more elegant.

Code Contest I'm not showing the entire body of code, but have a long script that pulls data from databases, cleans it up, and pushes it to an online database to power some of our analytical applications. The only thing I'm waiting on is resolution on running a Prefect agent on Windows as occasionally the Prefect agent will just crash. After that is resolved I'm going to deploy this to a virtual machine. If it is not resolved within a week or two, I might just have to figure out how to run this on a RHEL virtual machine within our firewall..

Code Contest I created a flow combining two things I have a strange fondness of: 1. Prefect flow run names 2. Craiyon ("DALL·E mini") images The flow takes the 5 latest flow run names and uses them as prompts for Craiyon. Though not really identifying any Prefect edge cases, it's an example of using: • Using native Python inside a flow • Using Prefect's logger • Mapping • .submit() • Interacting with PrefectFutures • Interacting with the API • Caching • Task concurrency limits --- Code here: https://github.com/taljaards/Prefect-2.0-code-contest

Code Contest Just two async subflows playing an infinite recursive game of random tic-tac-toe. Using the new Block primitive to manage shared state of multiple "Board" blocks (one for each game) between "nought" and "crosses" subflows. New games are started again by recursively calling the parent "tic_tac_toe" flow. Each move / square is a Prefect task. Rules: • Loser gets the starting move in the next game • Draw spawns two new games Subflow states: • Win = Completed • Lose = Failed • Draw = Cancelled You can keep track of the game tally between X and O by filtering flow run states. Open questions: • Are blocks threadsafe? Gist: https://gist.github.com/topher-lo/d2cf1124f68371716725869917d31aaf

Code Contest I am currently exploring the prefect 2.0 features (blocks, async, deployments, ui, schedules, etc.) and thought it would be interesting to do it in a fun way. This flow fetches random quotes from the internet using async/await and combines them with a random cat gif. The result is then send to a Slack channel using Prefect Slack Block. The user can then vote which one they like the most, of course counting starts at 0. The flow is scheduled to run every day through Prefect schedules. Currently it is only possible to send text to Slack, that’s why I provided a pull request which allows to send e.g. images or gifs using Slack blocks: https://github.com/PrefectHQ/prefect/pull/6200

I just created to show a use case of deployment in Prefect 2.0: Schedule your data workflow. This video covers the main concepts such as deployment, work-queue, and agent. If you are new to deployment, this video should make it clearer to you.

Code Contest At Empiric, we’re a IOT platform for industrial manufacturing. We utilize prefect as our distributed job scheduler. Our infrastructure utilizes AWS ECS, running prefect 2.0. The system resources are controlled via AWS CDK, making it easy for teams to launch their own stack of job orchestration system, or run on the central company infra. Below, one of our flows is responsible for performing vibration analysis on sensor data from machines out in the field. The flow can process multiple days of data from multiple sensors, and utilizes sub-flows for each individual sensor & day combination. For tasks, we abstract the logic into individual classes, allowing for a decoupling of the logic from underlying prefect tasks. This simplifies testing as well, as our unit-tests have no prefect related dependencies.

I like watching the flow running with this feature, looks like a spider's web:

made a simple webhook client to transform prefect webhook events into datadog events (also a shameless plug to maybe support generic webhooks in prefect 2.0 🙏 ) https://github.com/acookin/prefect-dd-webhook I've been using this to generate flow run metrics, and to alert when flows are not running as expected

I just published the article Detect Defects in a Data Pipeline Early with Validation and Notifications. This article covers how to: • Use Deepchecks to validate components in the research phase of your data science pipeline • Use Prefect to send notifications when a validation failed Source code.

I wrote a really small and compact guide on how to automatically trigger Jira Tickets on Task Failures: https://dev.to/codiefz/create-jira-ticket-on-prefect-task-failure-cj3 Following the docs I ran into some issues, mainly because some imports were missing, so perhaps it might be useful to some

We wrote about how we are using Prefect to orchestrate all the components and data pipelines in Endeavor's home grown machine learning platform. https://medium.com/@endeavordata/machine-learning-platform-at-endeavor-93ba88b66986 Would love to hear how others have used Prefect in similar ML use cases!

Hey folks 👋 I’ve just released

prefect_transfrom==0.1.6

. Docs che be found here. This release introduces a new Transform Block, which is now required to handle sensitive information to authenticate on Transform. Give it a try! Thanks @Andrew Huang for the guidance on implementing Prefect Block! 🙌

And

prefect_stitch==0.1.3

has a brand new block too!

StitchCredentials

, which can be used to handle Stitch access token by leveraging the Prefect Block concept 🦜 🦜 🦜 Docs can be found here.

@ale love how you’re already putting blocks to great use!

Hello Prefect community. 👋 Wanted to share a small code snippet I've been using to resolve this open issue: https://github.com/PrefectHQ/prefect/issues/5853 It's not technically a "fix" as I am creating new functions in the namespace via currying 🍛, but I think it's a cleaner solution compared to the current hack of using "deepcopy". @Jacob Danovitch @Jeff Carrico @Anna Geller Hope this helps others encountering this issue!

import asyncio
from prefect import task, flow

@task
async def print_x(x):
  print(x)
  await asyncio.sleep(1)

def build_subflow(name):
  @flow(name=f"subflow:{name}")
  async def subflow(x):
    await print_x(x)
  return subflow

@flow
async def parent_flow():
  futures = [build_subflow(name=x)(x) for x in ["a", "b", "c"]]
  await asyncio.gather(*futures)

If you like Airbyte, check out @Nate's latest blog post showing how to use our

prefect-airbyte

collection in Prefect 2.0

For those who just started with 2.0, explains what a flow is and how to use a flow in your project.

For anyone looking to get up and running with Prefect on Azure, I've published two tutorials over the past week (with more on the way!): • How to Run a Prefect 2 Agent in an Azure Container Instance • How to create Azure Blob Storage event-driven Prefect 2 flows with Azure Functions I also published a quick tutorial on how to run a Prefect agent as a systemd service on Linux.

If you’re looking to migrate your Databricks tasks from Prefect 1 to Prefect 2 with the prefect-dbt collection, check out this hot-off-the-press guide from @Andrew Huang! 🔥

Hello I am trying to integrate prefect cloud with Airbyte. But I am facing below error

ModuleNotFoundError: No module named 'prefect.tasks.airbyte'; 'prefect.tasks' is not a package

Have I missed to install any dependency ? I didnt see anything on airbyte/prefect pages.

Elevating this so it doesn’t get buried — thanks @Adebayo Adejare! https://prefect-community.slack.com/archives/CL09KTZPX/p1661533329261399?thread_ts=1661493823.803219&cid=CL09KTZPX

I noticed a few community members trying to run Prefect 2 on CentOS 7, which can be tricky because it comes with an old version of Python and a very old version of SQLite. CentOS 7 is tricky because it is old but still supported, so many enterprises still use it and consequently, you might need to run Prefect on it. So, I put together a quick tutorial that demonstrates a couple of ways to get Prefect up and running quickly on CentOS 7 (and Red Hat Enterprise Linux 7, since CentOS 7 is built from RHEL 7's codebase.)

What is a subflow? When should you use subflows and when should you use tasks? will answer those questions.

Hey all! 👋 I demo my two favorite new Prefect 2.3 features in the video linked here. Hope you find it helpful. 🙂

I wrote a guide to making deployments with a Python file instead of the CLI. Check out this post and please let me know if you find it helpful! 🐍 https://medium.com/the-prefect-blog/deploy-prefect-pipelines-with-python-perfect-68c944a3a89f

Hi everyone! I'd like to share with you a nice project orchestrated entirely with Prefect on top of Raspberry Pi. The project consists of an ML/AI-based football/score predictive model, running predictions for 100+ competitions across the globe, covering the entire Machine Learning end-to-end: from data acquisition, validation, model prediction/training and evaluation. I have also used the integrations with Slack (modified for Telegram) and Great Expectations for data validation/quality. In case you are interested, here is the link for the Telegram channel: t.me/alea_jacta_est_circe if you want to have a look to the predictions and results done on a daily basis with Prefect as the orchestrator engine. Hope you like it and enjoy it as much as I did integration Prefect with Raspberry Pi!

Just published - if you have some workflows that depend on events, such as when a new file arrives in a certain directory, this post may help you design your flow

A few days ago I asked about prefect+meltano in docker, and although it seems it wasn’t a common use case, I now have a working prototype if anyone is interested: https://github.com/radbrt/prefect_flows/tree/main/projects/jottings/meltano_flow It is in prefect 1 for now, but should be easy to port.

if you couldn't join today's PrefectLive, I highly recommend watching the recording