Hi all - Marvin is orchestrating a party 🥂 Our Slack community is growing quickly 🚀 and it would be great to be able to meet each other in a casual setting. We've found a virtual space that we enjoy using called Spatial, and we'd love to meet you all there. RSVP for our happy hour on November 18th from 730p 930p on Meetup. https://www.meetup.com/Prefect-Community/events/281892016/

Here's something to celebrate at the happy hour: A new release of Orion, Prefect's second-generation workflow engine, just dropped 👏 ! The highlight of this release is Executors, key building blocks that enable you to execute code in parallel, on other machines, or with other engines: Dask executor Orion first shipped with a Dask dask executor that could run simple, local clusters. This release marks feature parity with the existing Core Dask executor. You can create customizable temporary clusters and connect to existing Dask clusters. Additionally, because flows are not statically registered, we're able to easily expose Dask annotations, which allow you to specify fine-grained controls over the scheduling of your tasks within Dask. Subflow executors Subflow runs are a first-class concept 💪 in Orion. Subflow runs can have different executors than their parent flow run, which enables new execution patterns. Check out our multiple executor documentation for an example. Fixes & improvements * Updates to database engine disposal to support large, ephemeral server flow runs * Improvements and additions to the

flow-run

and

deployment

command-line interfaces * Clarification of existing documentation and additional new documentation * Fixes for database creation and startup issues Learn more in the latest release notes.

Hey everyone, a new release of P Core - 0.15.8 - just shipped 🎉. It's chock-full of new features and enhancements, many of which came from this community. Here are a few that we're particularly excited about: gcp Our very own @George Coyne worked with our friends at Square, @Damien Ramunno-Johnson and @Bradley Axen, on a new GCP Vertex AI agent and run config, allowing flows to be run serverlessly on GCP without having to manage GKE clusters or GCE instances. 📆 @Nitay J added support for rich scheduling via recurrence rules (RRules). RRules are the iCal spec for recurring events, used ubiquitously by calendar apps. You could use this same spec to create an ics file for a flow schedule in your personal calendar app. dbt @ale contributed a dbt Cloud run job task to the library so all you dbt users out there can trigger cloud runs via Prefect! Learn more about the other enhancements and fixes in the release notes. Special thanks to @Josh, Jonas Miederer, @Timo, and @Brett Naul for their contributions.

Hello everyone! Prefect Cloud and related services are currently being impacted by what appears to be a GCP load balancer issue affecting multiple sites and services across the internet; we have updated our status page here: https://prefect.status.io/ and are investigating the extent of the issue

Prefect Cloud is back online and accepting traffic; we are continuing to monitor the incident and posting updates on our status page (linked above)

👋 Hey all! We've been experimenting with livestreams lately and want to try something new. Come join @George Coyne 30 minutes from now (from 130p 230p Eastern) on Twitch as he writes flows live. Here's the link: https://www.twitch.tv/prefectlive Spam marvin duck in chat to give him energy.

👋 Hi there Prefect aficionados! I'm looking forward to the Community Happy Hour 🍻 happening tonight at 7:30p Eastern time. Sign up here and come meet the Prefectionists that you are interacting with every day such as @Anna Gellerand @Kevin Kho. We all look forward to chatting with you later tonight!

Hey there, we've got more news to share 🚀! After building Orion - an orchestration engine that was faster 🏃 and more dynamic 🏄‍♂️ than anything that came before - we found that existing graph visualizations couldn't handle its scale and flexibility, so we designed one from first principles, Radar:

We’re excited to share it with you! Please install the latest version of Orion

pip install -U "prefect>=2.0.0a"

, run some flows, and share your feedback in #prefect-community. We are continuing to design features for Radar, including the ability to monitor data, artifacts, and progress as the system processes, and we welcome your input.

👋 Hello perfect Prefect people! Let me put something on your radar. Go with the flow and join @Laura Lorenz's newest livestream happening on December 10th at 12:30p ET/9:30a PT. Note the earlier time - international folks, we heard you! Tune in to check out: 🌠 Orion update ↪️ A demo of subflows radar The new Radar UI If you like things that you enjoy, you're going to love it! Sign up here to attend.

Hello community! I’m excited to share that today we’re launching “Verified by Prefect,” a new initiative to ensure that key tasks in the library are up to date and continue to leverage new functionality and capabilities. The program will also make inter-product experiences are top notch for those in the program (documentation, sales, support, etc.). Launch partners are some of the most used and anticipated integrations in our ecosystem: dbt Labs, Fivetran, Great Expectations, Airbyte, Firebolt, PagerDuty and Monte Carlo. You’ll now see the badge below in the task library and can’t wait for you to try some of them out. (Note Monte Carlo and Firebolt are coming in the next 45 days). Please read the full blog here.

Hi everyone 👋 Don't forget to join @Laura Lorenz on Friday at 12:30 PM Eastern for a livestreamed demo of 🌠 Orion featuring subflows and Radar UI. Sign up on Meetup or join directly on Friday on .

Hello Prefect community! 👋 @Laura Lorenz is going live in 15 minutes and we'd love for you to join. To see 🌠 Orion 🌠 subflows and Radar UI in action, catch us on starting at 12:30p Eastern!

This week at Prefect Hey gang 👋 thanks for another awesome week in the Prefect Community! A lot happened: 🌠 Orion 2.0a6 release 🤝 Verified by Prefect launch marvin Marvin's first newsletter came out this Tuesday! For a summary of community content bi-weekly, you can sign up here! radar @Laura Lorenz's on Orion subflows and Radar UI Next week at Prefect 💨 For those interested in Airbyte, please join our very own @alex on the Airbyte community call next Tuesday Dec 14 at 3:30p EST as we demo our new integration. 📺 @nicholas takes the reigns for PrefectLive on Wednesday Dec 15 at 3p EST 🔥 @Chris White and @Jeremiah host another fireside chat on Friday Dec 17 at 4p EST where they'll give a glimpse into the future of Orion. Sign up on Meetup!

Airbyte/Prefect integration session starting now! We’ll review our integration and walk through an example. https://app.livestorm.co/airbyte/airbyte-community-call-08-ft-alexander-streed-from-prefect?type=detailed

Hey everyone! Users are currently experiencing issues logging into the UI, and we have updated our status page accordingly: https://prefect.status.io/ We believe the issue is with our upstream auth provider but are still investigating - the Cloud API is still healthy so scheduled work, queries, and triggering flow runs are all behaving as expected.

Howdy gang 👋 There was another great week full of ACTIVITIES, thanks for letting us be a part of your busy schedule. 🦑 On Tuesday @alex crushed it on the Airbyte Community Call (), thank you to the Airbyte team for letting us crash your party! Thinking we need an Airbyte emote ASAP. 📺 @nicholas helped schedule some Christmas gifts for Prefect employees live on Twitch. If you missed it, you can check out the ! There's some great Orion subflow and Radar UI examples in there. 🔥 We've got another Fireside Chat with @Jeremiah and @Chris White today from 4p-4:30p Eastern. Sign up on Meetup or join , and if you have any questions you'd like them to cover you can submit them here ahead of time! We have a hunch that 🎅 might need some help delivering presents next week. Be on the lookout!

Uh oh! 🎅 reached out to us and let us know that he needs help delivering presents this Christmas. Join his elf @George Coyne as he tests out the Prefect flow for making sure everyone has their presents this year. twitch.tv/prefectlive

👋 Hi everybody and Happy Holidays from all of us Prefectionists. I'm here to share our 2021 Growth Report, which highlights some of the most exciting things about this community including: Almost 8,000 Github ⭐ Tripling ↗️ our community size 2x contributor growth 10x task run🏃 growth on Cloud alone (!) Check out the full 2021 Growth Report here, which this entire community helped contribute to. We're incredibly fortunate to work with people like you every day, and we have some really exciting things in store for 2022.

👋 Hey folks! The most recent Orion release just shipped 🚀! On the heels of the recent rename of

Executor

to

TaskRunner

🏃, this release introduces

FlowRunner

🏃‍♂️, an analogous concept that specifies the infrastructure that a flow runs on. Just as a task runner can be specified for a flow, which encapsulates tasks, a flow runner can be specified for a deployment, which encapsulates a flow. This release includes two flow runners, which we expect to be the most commonly used: • SubprocessFlowRunner - The subprocess flow runner is the default flow runner. It allows for specification of a runtime Python environment with

virtualenv

and

conda

support. • DockerFlowRunner - Executes the flow run in a Docker container. The image, volumes, labels, and networks can be customized. From this release on, Docker images for use with this flow runner will be published with each release. Future releases will introduce runners for executing flows on Kubernetes and major cloud platform's container compute services (e.g. AWS ECS, Google Cloud Run). To learn more about flow runners and their relationship to deployments, check out their respective concepts docs themoreyouknow!

Hey all 👋 Prefect is celebrating its 4th birthday this week, and we wanted to send a special shoutout to this Pcommunity. Thanks to everyone for contributing, interacting and making this a special place to be. 🥂 Here is to many more years of making lives easier. Happy engineering!

👋 Hey everyone! Another week, another Orion release! Yesterday, we shipped the Logs release, another major milestone on Orion's continued evolution into a production ready tool 💪 . Orion's logs are designed to work exactly the way that you'd expect them to work. Our logger is built entirely on Python's standard library logging configuration hooks, enabling fine-grained customization of log levels for flows and tasks, including configuration for default levels and log message formatting. Orion captures logs for your flow and task runs by default, even if you have not started an Orion API server. You can view and filter logs in the UI, or access log records via the API or CLI. Learn more about this release in the release notes or check out the logging docs.

👋 Hey everyone, we’re very excited to share Prefect Core 1.0rc1, our release candidate for Prefect Core 1.0 🎉! Prefect Core has matured into a rock-solid, stable workflow management framework. We hope you’ll help us make sure it’s ready for the official 1.0 release by trying it out and letting us know if it breaks anything for you. 🛠️ Ready to give 1.0rc a spin? Since this is a prerelease, it will not be installed by default. Use 

pip install prefect==1.0rc1

 to try it out. The release is also available in a Docker image at 

prefecthq/prefect:1.0rc1-python3.8

. Highlights of this release include: • API keys are the way to authenticate with Prefect Cloud. Support for tokens has been removed. Check the API keys docs for details on moving to keys. • Flow 

Environments

 have been removed. Use 

RunConfig

 instead to define where and how a flow run should be executed. We deprecated 

Environments

 back in 0.14, but if you’re still using them, you won’t get a warning anymore. We have a guide on upgrading to 

RunConfig

 if you need it. • We’re dropping explicit support for Python 3.6. There are some breaking changes in this release. You can read more about them in the changelog. Otherwise, check out the 1.0 documentation preview or join the discussion on Github. Soon we’ll have topics on migrating to Prefect 1.0 as well. We’re excited about this milestone and thankful to have your support through the process. 🚀

Hey gang! 👋 Catch @Kevin Kho and myself on Twitch for PrefectLive at 3p Eastern today. We'll be making jokes, I'm working on a soundboard so I can interrupt Kevin with loud 🎶, and we'd love to interact with chat and take your questions. Oh yeah he'll do some cool stuff withP& dask, and train some scikitlearn models as well.

👋 Hi everyone! We’re super excited to announce that we have launched a P Discourse forum! You can sign up using the link discourse.prefect.io. Discourse makes many things easier for you than Slack can: • it’s easier to find relevant information through tags, categories, and a global search bar, • easier embedding of screenshots and code blocks from GitHub for troubleshooting, • syntax highlighting when you share code blocks with us (including logs), which makes discussing code that much easier, • easier quoting and cross-linking of topics and external resources, • much better categorization of content and support questions bringing more structure for everyone involved, • Google-searchable content, allowing you (in the near future) to google any question about Prefect and find a relevant Discourse topic with a solution to your issue, • and many more. You can think of Discourse as a knowledge base we can grow together as a community. Prefect Discourse is a great place to: • ask technical questions, • find the information you need when you need it, • get notified about the latest posts related to your favorite Prefect features and integrations, • learn more about Prefect and Orion, • find tutorials and code examples, • share solutions and tricks about using Prefect. To find out more about Discourse, check out this Getting Started with Prefect Discourse guide. We encourage you to sign up and say hi through the welcome topic.👋 ❗ Important note: Discourse does NOT try to replace Slack! We love chatting with all of you, and we’d love to continue doing so using Slack! To prove it, we have integrated a Discourse Slack app that allows us to transcribe any Slack discussion into a Discourse topic. To see it in action, check out this topic. We will continue answering all your questions wherever they might be coming from - Slack, Discourse, StackOverflow, GitHub issues - we are here to support you. But we encourage everyone to submit at least one question via Discourse, see if you like the format, and give us feedback. See you in Discourse and here in S!

👋 Hello all! There's a lot in it to be excited about in the latest P Orion release, 2.0a10, but we're particularly excited to introduce the new concurrent task runner and task run concurrency limits. Together, these features enable you to take advantage of Orion’s native

asyncio

(and

anyio

) support, harnessing the power of concurrency without complex software patterns such as thread pools. The concurrent task runner enables tasks that are blocked on input/output to yield to other tasks, whether your code is synchronous or asynchronous. When running a flow using a task runner that enables concurrent execution, or running many flows across multiple execution environments, you may want to limit the number of certain tasks that can run at the same time. For example, perhaps you want to ensure that, across all of your flows, there are no more than three open connections to your production database at once. You can do so by creating a “prod-db” tag and applying it to all of the tasks that open a connection to that database. Then, you can create a concurrency limit with

prefect concurrency-limit create prod-db 3

. Now, Orion will ensure that no more than 3 task runs with the “prod-db” tag will run at the same time. Previously, this feature was only available in a paid tier of Prefect Cloud, our hosted commercial offering. We’re very happy to move it to the open source domain, furthering our goal of making Orion the most capable workflow orchestration tool ever 💪. There is so much more in this release, including greater flow parameter flexibility, database migrations, CLI refinements, and much more. Read all about it in the release notes! 🚨 Breaking change: If you have previously installed any version of Orion, you'll need to delete existing database with

rm ~/.prefect/orion.db

or stamp it with

prefect orion database stamp

to successfully upgrade.

Hey there P gang! We've got a couple of livestreams coming up this month I wanted to highlight! Next Friday, February 18th at 3p Eastern, @Jeremiah and @Chris White will have another cozy 🔥 fireside chat to chat about the latest Orion developments, our 1.0 Prefect Core candidate and what's coming up next for Prefect. Sign up to attend here, or join the event directly on ! On Thursday, February 24th at 2p Eastern, Monte Carlo is hosting our very own @alex as we cover the intersection of data reliability and orchestration. You can sign up for that live workshop here.

👋 Hello Everyone!  We have been getting requests for infrastructure support and wanted to make sure everyone knows how to get in touch with the Prefect Professional Services Team!  The team marvin duck is available for general advisement or jumping in with hands on keyboard to increase your bandwidth! Here’s what we’ve been working on: • Build and deploy cloud infrastructure 🛠️ • Infrastructure design and implementation of Prefect execution environments • General architecture review and advisement 🏗️ • Deployment and modification of Prefect flows • Code reviews - 1:1 and in the GitOps process prefect duck • Technical trainings in support of adoption and integration of Prefect Please feel free to add time to our calendar or send us an email cs@prefect.io to discuss pricing and how we can help!  Happy Engineering! P

Hey there P community! 👋 The Orion orion project continues its steady march to becoming the greatest workflow orchestration engine ever. This week, it is one step closer to production ready with Kubernetes enablement. We've included a new Prefect CLI command, 

prefect orion kubernetes-manifest

, that you can use to automatically generate a manifest that runs the Orion API, UI, and agent as kubernetes deployments. With the Kubernetes flow runner, you can now run flows as Kubernetes Jobs. If you're running Orion in Kubernetes, you don't need to configure any networking. When the agent runs your deployment, it will create a job, which will start a pod, which creates a container, which runs your flow. Check out the release notes or the tutorial to learn more about running Orion and flows on kubernetes .