💡Spotlight on several new features💡 It's been a while since I posted here, so lots to catch up on. Over the last couple of months, we've introduced several new features both in our UI, api and in our core engine. Take a look at our changelog to see some of the cool features the team has been working on! Wishing you all a Happy New Year.

Prefect turns 3 tomorrow - come celebrate with us! Please join our team and community for a virtual happy hour tomorrow at 8pm ET / 5pm PT. See you there! https://spatial.chat/s/prefect-birthday-party

💡Spotlight on a new feature💡 Thank you to @Peyton Runyan for adding SQL Server to our task library. We now have three major relational database management systems included in our list of pre defined tasks. For anyone else that has an idea for a new task library and would like to contribute, here is a really great blog post by our very own @josh.

👋 Do you work for a 🚀 startup, 🎓 academic institution, 🌎 or non-profit? We’re running Lighthouse Partner programs for these three groups that include free access to Prefect Cloud. If you’d like to apply, please email hello@prefect.io. Thanks!

🚀 :marvin-duck: :marvin: We have Liftoff! Today we’re announcing Prefect’s Liftoff initiative - a huge series of announcements that will greatly advance our product. All of this is thanks to the incredible growth and enthusiasm of this community. Though we confess that sometimes it’s a challenge for our small team to keep up with all of you (😅) we are committed to giving everyone here a first-class and positive experience and a large part of Liftoff is aimed at ensuring we are able to do so. Here are a few of the items we’re previewing today: • Prefect’s Series A financing • Our product vision for a modern workflow API • Success-based pricing for Prefect Cloud • Partnerships with Microsoft, Slate, and Coiled • Three new community programs: Prefect for Academia, Prefect for Startups, and Prefect for Nonprofits One very important thing that we’ll announce soon: a new referral program that will let us give back to this community to properly thank our product ambassadors. As always, we appreciate your excitement and are grateful for your interest. Stay tuned in the coming weeks for full details on each of our Liftoff announcements. Happy engineering!

Prefect 0.14.7 has been released 🚀 Highlights!: • Expanded support for docker networks within the Docker agent • Simplified the use of Secrets with `GitHub`/`Bitbucket`/`GitLab` storage classes • Added new

Module

storage class, for referencing flows importable from a Python module • New tasks for working with Exasol Big thanks to all our community members who contributed this week 👏👏: • Alex P. • Marwan S. • Peter Roelants • Timo S. The full release notes can be found here: https://github.com/PrefectHQ/prefect/releases/tag/0.14.7

Hey folks! Data Council is doing a very short survey on what OSS Data tools people are using; the survey is an open response for 3 “open-source data projects” that you’re excited about. They plan to compile and share the results next month — if you’re so inclined, give it a quick look and don’t be shy about including your favorite workflow tool 😉 Direct link here: https://form.jotform.com/210275762605050

💡Spotlight on a new feature💡 Last night we released a new feature to our UI so that you can override your run config in real time. This is further enhancing the flow run configuration that we added to our core workflow engine last year. For those still looking to upgrade from <0.14.0, see our documentation on the best path forward.

Prefect 0.14.9 is out! 🚀 🚀 Highlights: • Introduced a new

CompressedSerializer

for compressing task results • Added configurable timeouts to the Prefect Client • Improved performance for all Prefect AWS integrations And last but not least, Fivetran introduced their first official orchestrator integration with a new Prefect task (the

FivetranSyncTask

); we’re excited to collaborate more with the Fivetran folks so give it a spin and let us know your feedback! Thanks to all our community contributors this week 👏: • Amanda Wee • Andrew Hannigan • Craig Wright • Nick Acosta • Timo S. The full changelog can be found here: https://github.com/PrefectHQ/prefect/releases/tag/0.14.9

Prefect 0.14.11 has been released 🎉 Highlights: • New

prefect server config

command that outputs a standalone

docker-compose.yaml

file that can be used without the prefect CLI. • New task library tasks for working with files/filesystems in

prefect.tasks.files

• Added

project_name

/`project_id` to flow run execution context in orchestrated flow runs Thanks to all our community contributors this week 👏 • David Zucker • Jacob Hayes • Joël Luijmes • Timo S. • Yogi Patel The full changelog can be found here: https://github.com/PrefectHQ/prefect/releases/tag/0.14.11.

Prefect 0.14.12 has been released 🚀 Highlights: • Complete redesign of the ECS agent to better support per-flow-run configuration and pre-registered ECS task definitions • Two new filesystem tasks Thanks to all our community contributors this week 👏 • Ben Fogelson • Marwan S. • Timo S. The full changelog can be found here: https://github.com/PrefectHQ/prefect/releases/tag/0.14.12

Additionally! Thanks to several community members (and our own @Michael Adkins), we have our first official Prefect Server Helm Chart release! 😛arty-parrot: 😛arty-parrot: The chart had been marked as "experimental" for the last few months, but today that flag has been removed. The Helm chart is now versioned, which pins each chart release to compatible releases of Prefect Server, the Prefect UI, and Prefect Core. Versions are published to a helm repo at https://prefecthq.github.io/server/. See the README for installation instructions. On Friday next week (3/19) at 4 EDT @Laura Lorenz will be doing a livestream of how to deploy this helm chart on Azure, see the meetup link for more information

Hi everyone! 👋 As some of you know, for the last 2+ months we’ve been piloting a new pricing model for Prefect Cloud and we’ve been pleasantly surprised by the wonderful reception by the community. Many of you have already explored this new way to join Prefect Cloud, through our expanded free tier. With feedback from community members, four ideas became central to what we call “success-based pricing”: 👍 Pay only for what you need  ✅ A generous free tier that includes 10,000 free runs every month (enough to power the majority of small business ETL use cases)  📉 Automatic volume discounts based on your usage  🚀 Committed usage plans for those looking for even deeper discounts or predictable pricing What makes it “success-based”? Pricing is based only on the number of successful task runs processed. We never charge for retries, failures, or even successful tasks that take less than a second to run. And we heard your feedback: all plans come with unlimited concurrency! Whether you’re on a free plan or an enterprise user, we want to make sure that Prefect Cloud is accessible to all in an aligned way. This simple, success-based pricing is available for all new users today, and existing users can choose to upgrade to a free plan from your team settings. Happy Engineering! https://www.prefect.io/pricing

Prefect 0.14.13 has been released 🚀 This release introduces two new CLI commands: •

prefect register

for easier registration of flows (docs). •

prefect build

for building flows without registration (docs). Combined, these expose a lot of new & interesting deployment options. One is automatic flow reloading. The

prefect register

CLI supports a

--watch

flag which will watch your source files, and auto-register them on save. This can be useful for development, but also exposes an airflow-like pattern where new flows can be deployed by adding them to a directory. We have several additional CLI improvements in the works, so stay tuned. As always, big thanks to all our community contributors this release 👏 : - Abid Ahmad - Alexandr N. Zamaraev - Jacob Hayes - Timo S. The full changelog can be viewed here: https://github.com/PrefectHQ/prefect/releases/tag/0.14.13 Happy Engineering!

Hello Prefect community! 👋 Join Laura this Friday (4 PM ET) for a livestream about data science on Prefect. This livestream will demo using Prefect in an end-to-end data science workflow, which will illustrate why Prefect is the tool of choice for workflow orchestration. I’ll send a reminder on the day of the event with a direct link as well - hope to see you there!

Hello everyone! Today we're happy to announce we've released version 0.14.17 🚀 Some highlights from the last few releases: - New

Git

storage allows you to use any

git

based service for flow storage (this includes deploy token/key authentication!) - Error messages have been improved in more than five places to help guide users to a resolution -

ShellTask

now works on Windows and supports more configurable logging -

Bitbucket

storage now supports cloud servers - A custom terminal state handler can now be placed on a flow, allowing users to customize the final state of their flow with access to the states of their flow's reference tasks - An architecture diagram has been added to the docs (https://docs.prefect.io/orchestration/#architecture-overview) - The required docker-compose version to run

prefect server start

has been lowered and documented - Environment variables can be passed to the

UniversalRun

config -

PostgresFetch

can return data column names -

prefect server start

can now connect to an separately managed postgres instance There's always more, see the full changelog at https://github.com/PrefectHQ/prefect/blob/master/CHANGELOG.md Thanks to our community contributors! - Alessandro Lollo - David Zucker - Greg Lu - James Lamb - Sean Talia - Jonathan Wright - Ben Fogelson - Gabriel Montañola Remember this list isn't exhaustive and for every person listed in it, there are many more who have helped report issues and written PRs that help move us forward even if they aren't merged. Thanks to everyone here in our Slack community!

Hi everyone! 👋 Today we’re introducing KV Store: managed metadata for Prefect Cloud flows! KV Store allows you to share small pieces of metadata in the form of key-value pairs across your team and flows. You can use it to track useful information and state across your team. Some example use cases we've come across include: 🔢 the last processed value 🤝 easily share state across flows ⚙️ custom configuration Read more about the feature here https://www.prefect.io/blog/introducing-the-prefect-kv-store Please note: you'll need to upgrade to Prefect Core 0.14.21 to get full functionality.

Hi folks -- today we announced Prefect’s Series B! :marvin: :marvin-duck: 🎉 Thank you all for your advice and encouragement - and yes, even your complaints - that helped steer the product and make it as useful as possible. This funding will let us focus on relentlessly delivering value to our users and customers, and we’re excited to do so for years to come. We’re turning everything up to 11, and you can read more about it on our blog! We’ll have many more exciting announcements for this group soon. Until then, happy engineering from our team to yours!

Hello everyone! The Prefect team will be performing infrastructure upgrades on Prefect Cloud this Saturday, June 19th, between 6:30pm UTC - 7:30pm UTC (2:30pm ET-3:30pm ET). We do expect intermittent periods of disruption during this time window; to stay up-to-date please follow along on our status page: https://prefect.status.io/pages/maintenance/5f33ff702715c204c20d6da1/60cb9ef4ffceea053471149d

Hi everyone! Yesterday, we released Prefect 0.15.0 🚀. This release improves the Prefect Core Python API for running and inspecting flow runs, exposing some powerful new patterns. We've added: • A new CLI for running flows, prefect run, which makes the transition from local execution to running with a Cloud or Server backend seamless • The ability to run flows without an agent, which lets you run your flow locally with the rich set of features enabled by the backend API. • New tasks, create_flow_run and get_task_run_result, to create subflows and retrieve data from them. • New objects, FlowRunView, TaskRunView, FlowView, and TenantView, which enable you to pull the most recent information about flow and task runs from the backend without writing GraphQL queries • A new suite of CLI commands for managing authentication with API keys, simplifying authentication while retaining backwards compatibility with tokens. For more information and examples of how to use these features, check out this blog post. For the full list of enhancements and features, see the release changelog. We're excited to see what you can do with them. If you have a question or feedback about this release, we want to hear it! As always, you'll find us #prefect-community channel. Special thanks to @Stéphan Taljaard and @Tenzin Choedak for contributing to this release!

This morning, Ben Lorica released a podcast with our very own @Chris White covering a lot of the motivation behind Prefect and starting to touch on the near-term roadmap (more to come on that very soon!). In addition, Ben and his colleagues are conducting a survey on the state of data engineering. Let’s make sure Prefect is well-represented! https://www.surveymonkey.com/r/DEStwitter

Hey there! We've been happy to see the enthusiasm for the major improvements made in the 0.15.0 release. With the help of some community contributors, we've kept the momentum going with the most recent patch releases, including this week's 0.15.3 release. Cumulatively, these releases include: • Enabled retrieval of secrets from private Vault instances with new

VaultSecret

task • Expanded configuration to expose deeper settings for Docker flow runs • Enabled loading of additional repository files with

Git

storage • Added documentation for querying account role based access control data • Added

--no-schedule

flag to

prefect register

to disable flow schedules when registered • Enabled recursive flow registration and builds through globs • Enabled calculation of evaluation parameters on the fly through new

evaluation_parameters

parameter for the

RunGreatExpectationsValidation

task • Increased robustness of flow runs to connection errors through improved exception handling Special thanks to @Gabriel Montañola, @Nelson Griffiths, Pawel Janowski, @Yueh-Han, and @Chris Ottinger for contributing to these enhancements!

We've been working on some exciting 😛refect: updates that we're looking forward to announcing 🔜. In the meantime, our team and community members have squashed 🐛 and made incremental enhancements in last week's 0.15.5 Release. Here are some highlights: • Python 3.9 support is stabilizing and we are now publishing Docker images e.g.

0.15.5-python3.9

. • While registering very large flows, it was possible to see timeouts or incomplete registration. Now, flow edges and tasks are registered in batches allowing for much larger flows. • Agentless flow runs now receive the environment variables of the process that starts them; this makes agentless runs easier to use in production. •

--idempotency-key

support has been added to

prefect run

•

prefect run

will now ignore schedules on local flow runs by default and only a single run will be performed. A

--schedule

flag was added if you’d like the command to go forever and run the flow on a schedule. • Several task library improvements have been made, including a new

SnowflakeQueryFromFile

task, improved logging in the AWS

BatchSubmit

task, improved runtime handling for MySQL Tasks, and updated authentication for the

DremioFetch

task. • Log limits have been updated to reduce errors when many large logs are sent. Note that this release includes several breaking changes: • In a secure-by-default effort, services run by the

prefect server

cli now listen to localhost by default and are not accessible from other machines; use

--expose

to allow connections from a remote location. • Prefect Server will need to be upgraded to version

2021.09.02

before flows can be registered from this version of Core. Special thanks to Deepyaman Datta, Henri Hannetel, Johnny Tirado, @Kathryn Klarich, @Tenzin Choedak, and Vincent Xue for contributing to this release :thank-you:!

Hi everybody 👋! We're going to have a huge week next week. If you want to see the next generation of Prefect tech in action, sign up for our livestream with @Laura Lorenz on October 8th at 4p Eastern/1p Pacific. Sign up on Meetup, or attend the Livestream directly on . See you there!

Hello everyone <!here>! I'm extremely excited to finally share what we've been working on the last few months: Prefect Orion, our second generation workflow orchestration tool. If you've ever asked about dynamic tasks, subflows, self-hosting Server without Docker, or almost any other common question over the last few years, the chances are you'll find the solution with Orion.  So much of what we sought to achieve was driven by this community's questions and feedback, and I can't emphasize enough how great it's been to collaborate with you all! Now that this is live, please peruse the docs, test out the initial alpha release and give us feedback!  We want you all to be as excited as we are for the possibilities here and help us shape Orion's near term roadmap. To get started: • read about the vision of Orion directly from @Jeremiah: https://www.prefect.io/blog/announcing-prefect-orion/ • begin building and testing your new workflows with our revamped documentation: https://orion-docs.prefect.io/

Hi everyone, hot on the heels of our Orion announcement, we wanted to get feedback from the community on how you’re using Prefect today, where you’re finding value, and how we can improve. It should take 3-5 minutes. And you’ll be entered in a drawing to win Prefect swag! https://prefect.qualtrics.com/jfe/form/SV_41MC3mkByz18Ok6

Hi everyone 👋! Hot on the heels of the Orion announcement, we have some great livestreams coming up that will feature Orion: -Today at 4p Eastern/1p Pacific @Laura Lorenz will host another spooky October livestream and take Orion for a spin. She'll highlight the new API, subflows and caching, as well as the killer new UI and deployment details. Sign up here to attend! -On Friday, October 22nd at 4p Eastern/1p Pacific Prefect CEO @Jeremiah and CTO @Chris White will host a Fireside Chat focused on Orion. Anyone in the community is invited to join and ask questions about whatever is on your mind. Sign up here to attend!

Hi all 👋 - more Prefect content coming in 🔥! Scaling Prefect with Dask is always a hot topic in the community. Our friends at Coiled are hosting a livestream with Prefect CTO @Chris White and Coiled CEO @Matthew Rocklin, covering how to scale everything from a single process to an entire auto-scaling cluster. Join them on 10/26 at 6p Eastern/3p Pacific by registering here.

Hi there Prefect community :marvin-duck:! Coming up in 2 short days on Oct 22 at 4p Eastern we'll be hosting a 🔥 Fireside Chat with @Jeremiah and @Chris White. If you haven't signed up yet you can do so here. The primary focus of this livestream will be Orion🌠, but any and all questions are welcome. If you'd like to submit a question ahead of time you can do so here. If you have any questions live, you can also use the Youtube chat to ask them - we'll be monitoring it throughout the stream. I can't wait to see you all there! Until then, happy Prefecting.