prefect-community #announcements

its beautiful A fresh look and feel The Prefect UI just got a fresh coat of paint! We've carefully updated colors throughout the UI to ensure a more cohesive and visually appealing experience. Whether you're a fan of the light or dark side (or switch between both), you'll notice our interfaces now shine brighter and feel more harmonious. Dive in and explore the new hues!

sonic 11

prefect spin 15

prefect duck 9

prefect rocket 11

prefect build 10

🦜 12

squirtle cool 2

wizard2 9

justabill

08/31/2023, 9:06 PM

🔍 Search and you shall find (Cloud only) You can now search any Prefect cloud workspace for anything it contains - flows, runs, blocks, automations, and more. Just hit the magnifying glass above the switcher or ⌘+K on your keyboard.

prefect logo 7

🔎 11

prefect rocket 11

😮 15

👀 1

prefect spin 1

justabill

08/31/2023, 9:11 PM

🧭 Reducing context switching costs With the new simplified workspace menu and workspace switcher, you can now switch between workspaces and the places within them with just a couple of clicks. Helpful links and resources are now just a click away too.

🦜 8

🔥 8

prefect spin 8

👍 1

justabill

08/31/2023, 9:12 PM

You can learn more about the many other fixes and enhancements in the release notes. Happy Engineering prefect rocket

🦜 15

sonic 14

prefect spin 1

justabill

09/14/2023, 8:40 PM

👋 Hey everyone! We're excited to share a bit of a sneak peak of an exciting new feature, Error Summaries enabled by Marvin AI. Whenever a flow run crashes or fails, Prefect Cloud can now produce a brief, clear summary of the error logs. Check it out:

🦜 5

🚀 9

marvin ai 5

prefect spin 1

justabill

09/14/2023, 8:40 PM

This is the first Prefect Cloud feature that is enabled by Marvin AI, our AI Engineering framework. Wherever you see the marvin ai cloud, Marvin is working under the hood. ℹ️ To enable this feature, the administrator of your organization must opt-in on the organization's Settings page.

marvin ai 9

prefect spin 7

prefect build 6

🦜 9

marvin ai cloud 14

justabill

09/14/2023, 10:03 PM

Today, we released Prefect 2.13.1! prefect rocket This release is check full of fixes and refinements that we hope make your life just a little bit easier. For example: you can now filter out subflow runs from the list on the Flow Runs page, so you can focus on your top-level flows with less clutter. themoreyouknow Check out the release notes for more information.

blob attention gif 10

🦜 10

prefect spin 7

its beautiful 1

🙌 18

Jeff Hale

09/20/2023, 3:41 PM

Hey all! 👋 If you or members of your team are looking to level up your skills with Prefect, check out the Prefect Associate Certification Course (PACC)! You’ll learn all about the latest and greatest Prefect features! We’ve got a few spots remaining in several upcoming remote sessions taught by Prefect staff. Grab a ticket today and let me know if you have any questions! 🙂

🧠 5

👀 5

🦜 3

prefect spin 1

Chris Reuter

09/21/2023, 1:00 PM

👋🏼 We recently released the .serve() method, which lets you deploy your flow straight from your code. Here’s @Jeff Hale

demoing this▾

with a simple script. Use the .serve() method for: • Deploying easily to remote infrastructure straight from your python script • Understanding the basics of deployments, quickly, and then configuring them for your unique scenario If you have any questions about .serve(), I’ll be starting a thread in #prefect-community where you can ask all about how to use it!

🙌 10

❤️ 7

prefect spin 6

wizard2 3

prefect build 4

🦜 4

🚀 3

🎉 1

squirtle cool 1

Sarah Krasnik

09/21/2023, 4:58 PM

Hi all 👋 Today, we published the most in-depth article we've ever had on where exactly Airflow fails when it scales. This is why our team is so motivated with the product we're building, and hopefully serves as a good parallel to existing Airflow users. Scaling Airflow is hard, and I finally put words on paper exactly why from personal experience. Any comments, reactions, or thoughts always welcome! https://www.prefect.io/blog/implications-of-scaling-airflow

gratitude spasibo 3

🦜 6

sonic 4

🙂 3

👍 4

💥 4

wizard2 6

👎 2

prefect spin 1

Jeff Hale

09/21/2023, 8:53 PM

Today, we released Prefect 2.13.2! prefect spin This release adds several key features and improvements: 🔒 Server-side enforcement of deployment parameter schemas to catch invalid parameters earlier (opt-in). 🚀 Enhanced deployment flexibility with pattern-based deploying. For example, deploy all deployments for a specific flow like this:

prefect deploy -n flow-a/*

🛠️ Various fixes, including Prefect variable resolution in the deployments section of

prefect.yaml

📚 Documentation updates and 🤝 new contributions from the community See the release notes for more information. launch

🙌 9

🚀 8

prefect spin 1

Will Raphaelson

09/21/2023, 9:12 PM

Hi all, with today’s release, Push Work Pools are now generally available in Prefect Cloud. 🌊 Push Work Pools eliminate the need to run a Worker or Agent. 📌 For users running flows on GCP Cloud Run, AWS ECS, or Azure ACI, you can now create work pools that securely “push” flow runs directly to your infra to run as containerized jobs. 💥 Get started with push work pools by checking out our Push Work Pools Guide or jumping straight into the app and creating one!

🙌 14

prefect spin 14

prefect build 10

aws 9

❤️ 4

🦜 3

🫡 3

gcp 9

🙂 4

🚀 4

azure 8

👏 4

💥 3

📌 1

👀 1

Jeff Hale

09/28/2023, 10:36 PM

Today, we released Prefect 2.13.3! 🚀 This release adds adds a number of new features and improvements: 🏗️ Configuration of a work pool’s base job template via the CLI ⚙️ Ability to customize your default flow runs view in the UI gcp A VertexAI Work Pool/Worker option 🔍 Display

pull_steps

on Deployments page in the UI 📝 Documentation updates and 🤗 new contributions from the community See the release notes for all the details. prefect rocket

🙏 7

🚀 8

🔥 9

👍 7

🙂 7

🦜 11

🙌 8

🎉 7

Chris Reuter

10/03/2023, 5:28 PM

👋 Hi all! I'm looking for some good examples to use in our onboarding emails specific to automations. If anyone here has an interesting or novel use of automations specifically, I'd be interested to hear it. Feel free to DM me if you'd be willing to share thank you

Jeff Hale

10/05/2023, 10:00 PM

We just released Prefect 2.13.5! prefect spin This release introduces a few new features: ⚡ You can now load and serve flows from a git repository! With the new

flow.from_source

method, you can specify a git repository and a path to a flow file in that repository. This method will return a flow object that can be run or served with

flow.serve()

. 🧪 Experimental Pydantic 2 compatibility! We’re working eagerly toward having Prefect installable with either

pydantic<2

pydantic>2

. As a first step toward compatibility, we’ve ensured that Prefect’s use of Pydantic is isolated from your use of Pydantic in as many ways as possible. As always this release includes lots of bug fixes 🐛, documentation updates📃, and community contributions🤝! See the release notes for more details. prefect build

🙌 14

💯 5

🚀 5

💥 3

Sarah Krasnik

10/10/2023, 8:08 PM

☀️ Prefect will be in San Diego for Coalesce in just one short week! We've got some great representation, with both Jeremiah and Chris White there with many other wonderful folks. • We're at booth 222. Stop by, or schedule a specific time to chat with us. • Conferences have talks during the day, and after hours events—but what about happy hours? We're hosting a HH Tuesday @ 4pm PT. Swing by. • Yours truly is giving a talk on Thursday @ 10am PT. Calling all Survivors fans or the Survivor curious - I'm so excited for y'all to hear it! Find all relevant Coalesce info here: https://www.prefect.io/prefect-coalesce-2023

🏄‍♂️ 9

prefect rocket 11

prefect build 6

dbt 8

👍 1

Serina

10/12/2023, 9:23 PM

Prefect 2.13.6 has been released! prefect rocket This release includes a few exciting enhancements via additional Prefect settings: ✖️ We’re enhancing the reliability of flow run cancellation, which you can try out by enabling the

PREFECT_EXPERIMENTAL_ENABLE_ENHANCED_CANCELLATION

feature. When enabled, you can cancel flow runs where cancellation can fail, such as when your worker is offline. 📦 The new

PREFECT_DEFAULT_RESULT_STORAGE_BLOCK

setting allows users to set a default storage block on a work pool or via job variables for a deployment. It’s now easier to swap result storage locations by updating the environment in the UI or your

prefect.yaml

file, eliminating the need to alter your flow source code. See our release notes for more information. prefect spin

✨ 12

upvote 5

P 6

🚀 8

prefect build 2

Chris Reuter

10/17/2023, 3:22 PM

Data engineers orchestrate dbt dbt in an infinite number of ways, across a mix of self-hosted and managed offerings. Here's a tutorial on a few patterns that we see, featuring a new library courtesy of our partners at Dataroots called

prefect-dbt-flow

that displays dbt model execution in the Prefect timeline view. if you're in San Diego at dbt Coalesce this week make sure to stop by Booth 222 and grab one of these cool fuchsia ducks! We'll see you there dbt prefect rocket

dbt 8

prefect rocket 7

sonic 4

👍 6

squirtle cool 5

Jeff Hale

10/19/2023, 9:10 PM

Prefect 2.13.8 has been released! 🚀 This release introduces an improved Python deployment-creation method for flows that require expensive infrastructure.

flow.deploy

allows you to easily create a Docker image with your flow code baked in and use that image with the work pool of your choice. Updating a Python serve script to a deploy script is as simple as changing

serve

deploy

, providing a work pool to deploy to, and providing a name for the built Docker image. See the 🧵 for an example. As always, this release includes 🐛 bug fixes, 🏖️ quality-of-life enhancements, 📝 documentation updates, and 🤝 new contributions from the community! See the release notes for more information. prefect build

🚀 4

prefect spin 3

prefect logo 10

🙌 13

party deploy 3

Jeff Hale

10/25/2023, 7:44 PM

We’re pleased to announce the release of Prefect 2.14.0 and the inaugural release of the light-weight Prefect Client:

prefect-client 2.14.0

. prefect rocket Things to know about Prefect Client: 1. Installed size (with dependencies) is less than half the size of Prefect. 2. It’s a stripped down version of the Prefect package without a server, UI, or CLI. Many dependencies are removed. 3. You can have both

prefect

and

prefect-client

in the same environment (for example if you thought

prefect-client

would meet your needs but found you needed to install

prefect

for additional functionality). 4. You import objects from

prefect-client

exactly like you do from

prefect

. For example:

from prefect import flow

. 5. The Prefect Client is built out of the prefecthq/prefect GitHub repo and we plan to keep the versions of Prefect and Prefect Client in sync (the latest versions of both are currently 2.14.0). 6. We do not currently publish a Docker image that uses the Prefect Client.

prefect-client

is available today on PyPI, and can be installed with

pip install prefect-client 🚀

Please share any feedback about the Prefect Client in Slack. If you find bugs or desire additional features, feel free to open an issue in the Prefect GitHub repo. We plan to continue development it to make it as functional and light-weight as possible. This release also includes bug fixes 🐛 and documentation 📃 updates. See the release notes for all the details. 🔍

prefect spin 14

🚀 8

🎉 4

Will Raphaelson

11/02/2023, 3:44 PM

prefect spin Hello all - today, we’re pleased to announce team management in Prefect Cloud. Enterprise tier plans can now streamline workspace access and object-level access controls through teams of users and service accounts. Admins can either administer teams manually, or integrate with their IDP through SCIM to have their IDP teams automatically reflected in prefect cloud. 🫂 Give it a try by navigating to Account -> Teams, and let us know what you think!

👍 5

👀 5

📌 5

🙌 7

💥 5

🚀 4

Will Raphaelson

11/02/2023, 10:48 PM

👋 Prefect 2.14.3 has been released! 🚀 This release includes deployment status monitoring in cloud, storage options for

flow.from_source

, and a host of bugfixes and docs improvements. Deployment Status Monitoring 🚦 • Track deployment status in the Prefect UI • Emit Prefect Cloud events for deployment status changes, with automation triggers available Expanded Storage for
flow.from_source
🗂️ • Load flows from remote storage locations like S3 and Azure Blob Storage using

fsspec

protocols Enhancements and Fixes 🔧 • Skip Docker builds with

flow.deploy

• Fixes for cancellation cleanup, Pydantic V2 models in flow signatures Documentation and Guides 📘 • Adds documentation for Systemd daemonization, global concurrency clarification, and Prefect Cloud teams prefect spin See the release notes for more details!

🙌 10

🦜 8

catjam 12

🙂 4

❤️ 4

⭐ 4

🎊 3

🚀 8

Jeff Hale

11/09/2023, 10:51 PM

Today we released Prefect 2.14.4. 🎉 The highlight of today’s release is a new and improved flow run graph with dependency layout! 😍 • We rebuilt the graph from the ground up for better performance. Larger flow runs are now displayed much more smoothly. • We added three new layout options: two non-temporal options to provide a clearer picture of the dependency paths and an option to facilitate easy comparison of run durations. • The x-axis can now be independently scaled for temporal layouts. You can adjust the x-axis in the graph settings or with the new keyboard shortcuts - and +. • We included additional small bug fixes, including the display of cached tasks. We also made a number of improvements to the deployment creation process! 🚀 • Improved tag handling on

DeploymentImage

for

.deploy.

• Allow

image

passed into

.deploy

to be optional if loading flow from storage. • Ensure client avoids image builds when deploying to managed work pools. As always, bugs were squashed and documentation was improved. 🐞📖 Check out all the details in the release notes. prefect rocket

🚀 11

🎊 7

🦜 6

marvin duck 6

👏 3

❤️ 7

👀 2

prefect yellow 19

Jeff Hale

11/13/2023, 5:52 PM

Hey all! 👋 We are excited to announce that the Prefect Terraform Provider for Prefect Cloud has moved out of beta with the release of version 0.1.0! • Check out the Getting Started guide and other documentation in the Terraform Registry. • Refer to the GitHub repo to learn what’s covered, make a feature request, and see how you can get involved. Happy engineering! prefect build

🙌 17

prefect spin 13

🎉 7

❗ 13

Jeff Hale

11/16/2023, 11:57 PM

Howdy! 🤠 Prefect 2.14.5 has been released! Here are the highlights: 🧱 You can now integrate storage blocks seamlessly with

flow.from_source

for synchronizing credentials and configurations. See the example in the image below. 🎛️ We’ve added two new settings enhancements: • A client-side setting for specifying a default work pool. • Host and port settings for the worker webserver are now customizable. 📖 We’ve updated our work pool-based deployment guide to include information on creating deployments with

.deploy

. The release contains a number of other 🐞 bug fixes 📚 documentation updates 🤝 new contributions from the community Check out all the details in the release notes! prefect spin

🔥 13

🦜 8

typingcat 7

marvin duck 8

ultra fast parrot 7

prefect rocket 11

justabill

11/22/2023, 9:46 PM

This Thanksgiving, we're feeling grateful for this community, which is the source of so much feedback, motivation, and love. Thank you for using Prefect! 💙 Today's 2.14.6 release is a product of that motivation and feedback. Here are a couple of our favorite improvements:

justabill

11/22/2023, 9:47 PM

View the next run for a deployment at a glance 👀 You can now see a deployment's next scheduled run right at the tope of the runs tab in the Prefect UI! See the following pull request for implementation details: — #11230

🙌 5

🦜 3

prefect spin 2

justabill

11/22/2023, 9:48 PM

Automatic project configuration for Cloud Run push work pools gcp Push work pools in Prefect Cloud simplify the setup and management of the infrastructure necessary to run your flows, but they still require some setup. We've enhanced the

prefect work-pool create

CLI command to automatically configure your GCP project and set up your Prefect workspace to use a new Cloud Run push pool immediately. You can create a new Cloud Run push work pool and configure your project with the following command:

prefect work-pool create --type cloud-run:push --provision-infra my-pool

See the following pull request for implementation details: — #11204

🙌 5

🎉 3

⚡ 4

justabill

11/22/2023, 9:50 PM

For more information about today's release, check out the release notes. Happy Thanksgiving! 🦃

🙏 3

🦃 5

justabill

11/30/2023, 10:01 PM

We're continuing to make deploying flows with Prefect easier with today's 2.14.9 release! prefect rocket Prefect now supports Automatic infrastructure provisioning for ECS push work pools aws You can create a new ECS push work pool and provision infrastructure in your AWS account with the following command:

prefect work-pool create --type ecs:push --provision-infra my-pool

Using the

--provision-infra

flag will automatically set up your default AWS account to be ready to execute flows via ECS tasks:

Copy code

╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Provisioning infrastructure for your work pool my-work-pool will require:                                         │
│                                                                                                                   │
│          - Creating an IAM user for managing ECS tasks: prefect-ecs-user                                          │
│          - Creating and attaching an IAM policy for managing ECS tasks: prefect-ecs-policy                        │
│          - Storing generated AWS credentials in a block                                                           │
│          - Creating an ECS cluster for running Prefect flows: prefect-ecs-cluster                                 │
│          - Creating a VPC with CIDR 172.31.0.0/16 for running ECS tasks: prefect-ecs-vpc                          │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Proceed with infrastructure provisioning? [y/n]: y
Provisioning IAM user
Creating IAM policy
Generating AWS credentials
Creating AWS credentials block
Provisioning ECS cluster
Provisioning VPC
Creating internet gateway
Setting up subnets
Setting up security group
Provisioning Infrastructure ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 0:00:00
Infrastructure successfully provisioned!
Created work pool 'my-pool'!

If you haven't tried an ECS push pool yet, now is a great time! Don't fret, Azure users; we will add support for Azure Container Instances push work pools soon! soon horse See the release notes for information on the other enhancements and fixes in today's release. Happy engineering! marvin duck

prefect build 12

catjam 6

🚀 8

typingcat 6