Hi everyone! 👋 We've just released Prefect 2.6.8 prefect2 cloud2 This release includes new agent CL options, including the

--prefetch-seconds

flag, which is especially useful when interacting with serverless infrastructure. Imagine that you want to schedule a specific flow to run at 9 AM, but it roughly takes a minute to provision a serverless container. With this new CLI flag, your Prefect agent will begin flow run submission earlier than scheduled (in the example below, 60 seconds earlier) to ensure your container starts on time:

prefect agent start --prefetch-seconds 60 -q default

The release includes many bug fixes and a new

prefect-vault

collection allowing you to interact with Hashicorp Vault - big thanks to Pavel Chekin for contributing it gratitude thank you As always, you can find more details in the release notes. Happy engineering! prefect duck

Hi everyone! 👋 Prefect 2.6.9 release has just shipped! prefect2cloud2 It includes several fixes and enhancements to the logging experience. 🪵 You will now be able to: • see Dask worker logs in the Prefect UI dask • configure logging more easily in tests and CLI commands (a.o. when starting the Orion server) • configure logs to be properly formatted in JSON using

orjson

serialization (this is important for logging to external systems such as Datadog - more on that soon!) • avoid duplication when configuring logging e.g. to configure logs to be JSON-formatted, you can now use

PREFECT_LOGGING_HANDLERS_CONSOLE_FORMATTER="json" python myflow.py

Apart from that, the

prefect cloud login

CLI has been given a serious upgrade. ✨ For a detailed description of that new Cloud CLI login UX, and to see a full list of fixes and enhancements, see the release notes. Happy Engineering! P

Hi Prefectionists! P It's only been a little over a week since the last release, but there are a lot of new features in Prefect 2.7, which was released just now. prefect2 cloud2 This release includes: • Cancellation of flow runs • Pause and resume of flow runs • Logging of print statements • Agent-level flow run concurrency limits • Notification blocks for PagerDuty and Twilio • Improved reporting of crashed infrastructure • Improved handling of failures during HTTP requests • ... and dozens more fixes and enhancements! Thanks a bunch to our five external contributors! 💙 Also, a massive shoutout to @Zanie, who implemented several features of this release, coordinated the entire release work and even wrote most of this announcement and the release post. Check out the release blog post and reach out if you have any questions. Happy Engineering!

👋 Hi folks - Will here from the Product Team at Prefect! POur mission here at Prefect is to Coordinate the World's Dataflow. As part of this mission, we’re excited to announce the first major milestone of our investment into dataflow observability: Automations. As valued members of our community, I’m excited to invite you all to access Automations in its beta phase. ⚡ Automations in Prefect Cloud enable increased flexibility, control, and responsiveness of your dataflow by allowing you to configure Triggers and Actions in response to events — or lack thereof! -- from across the stack. This forms the foundation of the reactive dataflow that we’ll be rolling out in 2023. 🎏 For our flagship release, we’ve prioritized features to: • Configure Triggers based on Flow Run state changes (or lack thereof) and Work Queue health. • Execute Actions to run/cancel Flow Runs, pause/resume schedules, pause/resume Work Queues, and send beefed up context-aware notifications. 📈 This feature set will be a springboard for a wealth of exciting enhancements to Prefect over the next year, so your thoughts and feedback will be immensely valuable. 🫂 If you’re interested in joining the beta, react with your favorite emoji and I’ll add you to the program!

Hi everyone! 👋 We've just released Prefect 2.7.1 prefect2 cloud2 This release adds: • a task run concurrency limits UI page - you can now add new task run concurrency limits and see how many slots are taken at any time • bulk deletion of flows, deployments, and work queues from the UI •

PREFECT_EXTRA_ENTRYPOINTS

setting for module injection to allow registration of custom blocks (e.g. during CI/CD) • improved timeout handling on the

KubernetesJob

block - this timeout is now disabled by default (i.e.

job_watch_timeout_seconds

is set to

None

) • fix for the

prefect agent start --prefetch-seconds

command • fix for the flow run URL from notification blocks ...and several more intricate enhancements and fixes you can read about in the release notes. Big thanks to

t-yuki

and

padbk

for their first contributions. gratitude thank you Happy engineering! prefect duck

👋 Hi all - Will here again from the Product team. I’m excited to announce that we’re moving Automations into open beta. ⚡ Automations in Prefect Cloud allow for increased flexibility and control of your data stack by allowing you to configure triggers and actions. Using triggers and actions, you can automatically execute critical workflows such as kicking off flow runs, pausing deployments, or sending custom notifications (coming soon!) in response to real-time monitoring events. ❗ Starting today, you’ll see Automations in your workspace sidebar. Give it a try and let us know what you think. 🤖 Happy automating!

Hi Prefect Community! marvin We'd like to introduce you to @ask-marvin, our new GPT3-powered finely tuned chat bot where you can ask them any questions in #ask-marvin channel. Marvin is still acquiring knowledge and understanding, despite their advanced intelligence and vast experience. Big shout out to @Serina , @Nate, and @Rob Freedy for their incredible engineering skills building Marvin. Here's a few words from Marvin:

S The Prefect Slack community has grown by leaps and bounds (now over 23k people!). 🌎 Today we created geographic-based channels so folks could connect with other people nearby and be kept in the loop for community events. We’ve started with locations where we’ve had Prefect Associate Certification courses recently and where we have lots of users. Feel free to join a geo channel if there’s one for your area. 🙂 If there’s isn’t yet a channel for your area, add a comment to the community thread. If someone has already proposed your area in the thread, please give it a 👍. #geo-nyc #geo-bay-area #geo-dc #geo-chicago #geo-boston #geo-london #geo-berlin #geo-texas #geo-seattle #geo-colorado

Hi everyone! 👋 Prefect 2.7.2 is here prefect2 cloud2 This release extends the recently added *pause*/*resume* feature with rescheduling of paused flow runs. You can now pass the

reschedule=True

argument to the

pause_flow_run

utility to gracefully stop the infrastructure process and put the flow run back into a

Scheduled

state. This way, you can free up the infrastructure resources instead of keeping the original flow run process running while waiting for approval.

💡 Note that: this kind of rescheduling pauses works only with flow runs created from deployments and requires results persistence using either the

persist_result=True

option on the flow decorator, or by setting it globally:

prefect config set PREFECT_RESULTS_PERSIST_BY_DEFAULT='True'

You can now also manually pause a flow run in progress using the new Pause button directly from the flow run UI page (see the screenshot below). Resuming a run can be performed also from the UI, or from code:

from prefect import resume_flow_run, pause_flow_run

id_ = "pasy_your_flow_run_uuid_here"
pause_flow_run(id_, timeout=4200, reschedule=True)

# once you're ready to resume (you can do it from the UI, too):
resume_flow_run(id_)

Other enhancements and improvements in this release: • Two new Prefect Collections: prefect-kubernetes kubernetes and prefect-bitbucket 🪣 • Task Run Concurrency UI page now includes details and shows active task runs currently using allocated concurrency slots (see the second screenshot) • New Opsgenie notifications block - kudos to Julien Lutran for contributing! gratitude thank you • Fixes to imports and their performance • New

SecretDict

block field that allows to obfuscate nested values in a dictionary on blocks such as

GcpCredentials

or

FugueEngine

• If you have a flow call (e.g.

myflow()

) in your flow script, not protected by an

if __name__ == "__main__:

block, any run from that deployment will now work without giving you a warning about that unprotected call • Several documentation improvements incl. new

with_options

usage examples, recipe and tutorial references, Automations docs, and more! For all enhancements and fixes, check the release notes - it even includes a Marvin joke. marvin duck Happy engineering! 🎄prefect duck

👋 See you all soon! https://prefect-community.slack.com/archives/C036FRC4KMW/p1671572581358209

For those who missed it, here is a holiday stream. 🎄🎅 We wish you all Happy Holidays and a great start to the New Year! 🎇

Prefect 2.7.4 has been released! 🚀 Highlights ✨*:* • Improvements to retry delays: multiple delays, exponential backoff, and jitter • Increase in the default thread limit from 40 to 250 • New `prefect-google-sheets` collection - big thanks to community member @Stefano Cascavilla for creating it! Feel free to head on over to the repo and give it a ⭐️ As always, see the release notes for details. Thank you to everyone for your feedback and especially to @devanshdoshi9 and @quassy for their first contributions! Happy holidays! ☃️

Prefect 2.7.5 has been released! 🎉 Highlights • Schedule flow runs from the CLI with

--start-in

or

--start-at

e.g.

prefect deployment run foo/test --start-at "3pm tomorrow"

• Get the logs for a flow run using

prefect flow-run logs <flow run UUID>

e.g.

prefect flow-run logs 7aec7a60-a0ab-4f3e-9f2a-479cd85a2aaf

Collection updates • Added

BigQueryWarehouse

block in

prefect-gcp

v0.2.1 • Added

AirbyteConnection

block in

prefect-airbyte

v0.2.0 • Added dbt Cloud metadata API client to

DbtCloudCredentials

in

prefect-dbt

v0.2.7 As always, see the release notes for details. Thank you to everyone for your feedback and especially to @ohadch and @mohitsaxenaknoldus for their first contributions! Happy New Year! 🥳

Announcement only for Cloud 1.0 users that created auth tokens before May 2020: Last night (Wednesday, January 4th) we were notified by CircleCI of a security alert. The security alert does not indicate that Prefect was impacted, but strongly recommends that all CircleCI users rotate their secrets. As a consequence of this rotation, we will no longer be able to verify auth tokens that were generated prior to approximately May 2020. Therefore, Prefect Agents that use legacy “Runner” or “Tenant” tokens generated before May 2020 will stop working and will need to be provided with new API keys to continue running flows. Users who adopted Prefect API Tokens or API Keys after May 2020 should not be affected.

Hi everyone, happy Friday! 👋 Prefect 2.7.7 is here prefect2 cloud2 The API reference documentation has been completely rehauled with improved navigation and samples. You can see the REST API documentation on Prefect Cloud, as well as rendered in our open-source documentation. We've also improved the parsing and rendering of reference documentation for our Python API. See the tabular @flow decorator docs as an example. This release also includes lots of enhancements related to blocks: • PR 7862 added

validate

flag on the

Block.load

method to facilitate block document migration, and PR 8056 allowed schema migration of block documents using

Block.save

• when you register new block types with the

prefect block register

CLI command, you will now get a link to the UI Blocks page to easily create new blocks (8017) • when you try to save a block with an invalid name, you'll see better error messages (8038) Apart from that, there are improvements and fixes to (among others!): • flow timeouts (7993) •

prefect cloud login

(8034) • Kubernetes job (8018) and its documentation (8044) •

SecretDict

serialization (8074) • annotations (7263) and their serialization (8037) • task run cache key size (7275) • RRule schedule length - there is a limit now (7762) • deployment troubleshooting - we added a log message when Prefect downloads flow code from storage and from which storage location (8075) • the result factory, which now avoids creating unnecessary DB connections (8072) • documentation - lots of improvements here, including not only docs but also a new parser and renderer (7855) For all enhancements and fixes, check the release notes. Happy engineering and have a great weekend!prefect duck

Prefect 2.7.8 has just arrived! prefect2 cloud2 This release includes several highly requested features, including: 1. Flow run timeline view - apart from the radar view showing the flow structure and dependencies, you can now see how your task and subflow runs progress over time, making it easier to track concurrent execution and troubleshoot performance bottlenecks. This feature is currently in beta, and we have many improvements planned in the near future! We're looking forward to your feedback. 2. New argument

refresh_cache

on the

task

decorator - this allows you to rerun the task and refresh the cached result. You can think of it as manually invalidating the cache for a given task when needed. You may also temporarily change your Prefect Setting to refresh all tasks without changing your code using the command:

prefect config set PREFECT_TASKS_REFRESH_CACHE=true

3. Support for memory limits and privileged containers on the

DockerContainer

infrastructure block. Apart from that, there are improvements to client retriable exceptions, documentation,

orjson

serialization, and more. As always, check the release notes for more information. Big thanks to

toro-berlin

,

muddi900

,

ddelange

, and

Ewande

for making their first contributions! P

👋 Hi folks - Will here from the Product Team at Prefect! About a month ago, I announced the beta release of our first major dataflow observability feature, Automations. Today, I’m excited to promote that feature to GA across all Cloud workspaces. ⚡ Automations in Prefect Cloud enable increased flexibility, control, and responsiveness of your dataflow by allowing you to configure Triggers and Actions in response to events — or lack thereof! -- from across the stack. This forms the foundation of the reactive dataflow capabilities that we’ll be rolling out in 2023. 🎏 For our flagship release, we’ve prioritized features to: • Configure Triggers based on Flow Run state changes, Work Queue health, or custom criteria via the API. • Execute Actions to run/cancel Flow Runs, pause/resume schedules, pause/resume Work Queues, and send beefed up context-aware notifications with support for Jinja templating. 📈 This feature set will be a springboard for a wealth of exciting enhancements to Prefect over the next year, so as always, let us know what you think. 🫂 Happy automating! -WR

Prefect 2.7.9 is here! prefect2 cloud2 This release ships with multi-architecture development Docker images! docker ship This means that you can now pull both

linux/amd64

and

arm64

-based development images directly from DockerHub, for example:

docker pull prefecthq/prefect-dev:main-python3.10 # default: linux/amd64
docker pull prefecthq/prefect-dev:main-python3.10 --platform linux/arm64

There is also a new Prefect OpenAI collection for all AI fans! We've also: • G added a new interface to filter

.git

files when using the

GitHub

block • terminal added the

--head

and

--num-logs

flags to the

flow-run logs

CLI: ◦

prefect flow-run logs run_uuid --head

will give you 20 log messages for a given flow run UUID ◦

prefect flow-run logs run_uuid --num-logs 5

will give you 5 log messages for a given flow run UUID • 🐛 fixed several bugs, a.o. one for blocks and one for Windows subprocesses • 📚 added a new recipe contribution documentation page • aws added a new AWS Chalice recipe Big thanks to ddelange for consistently working on contributing several PRs to add the

arm64

-based Docker images as well as to

mj0nez

, and

ohadch

for contributing to Prefect!P For all enhancements and fixes, check the release notes. Happy engineering!prefect duck

Good Afternoon & hi everyone! Today we are announcing that legacy Prefect Cloud 1 accounts will be frozen for Free tier users on May 1st 2023, and for Starter & Standard tier users on monthly plans on July 15th 2023. This means that your account and tenant will become read-only and your flows & tasks will no longer run. Prefect 2 has seen exciting development advancements over the past year, and Prefect Cloud 2 is available with a free tier and a simple pricing model based on users & workspaces. To help our remaining users as they transition, we are providing upgrade support to all users via written guides, videos and help from real Prefect engineers. To get upgrade support, visit https://upgrade.prefect.io/. Note that this freeze only pertains to Free, Starter and Standard users of Prefect Cloud 1 on monthly plans, available at cloud.prefect.io. Enterprise users, those with annual agreements and accounts on Prefect Cloud 2 are not impacted by this announcement.

Hi everyone! Prefect 2.7.10 has been released! prefect2 cloud2 We're excited to announce an upgrade to our flow run cancellation feature, resolving many common issues. When cancellation is requested, the agent now sends a termination signal (SIGTERM) to the flow run infrastructure. Previously, this signal resulted in the immediate exit of the flow run. Now, we detect the signal and attempt to shut down the process gracefully. This gives the run an opportunity to clean up any resources it is managing. If the flow run does not gracefully exit in a reasonable time, it will be killed. More details about cancellation improvements: 1. We improved how we handle runs in the process of cancelling -- when a run is cancelled, it's first placed in a "cancelling" state, then moved to a "cancelled" state when cancellation is complete. 2. Previously, concurrency slots were released as soon as cancellation was requested. Now, the flow run will continue to occupy concurrency slots until a "cancelled" state is reached. 3. We added cleanup of tasks and subflows belonging to cancelled flow runs. Previously, these tasks and subflows could be left in a "running" state. This can cause problems with concurrency slot consumption and restarts, so we've added a service that updates the states of the children of recently cancelled flow runs. Additionally, from this release on, we'll be publishing multi-architecture Docker images to DockerHub not only for development images, but also for production images! docker ship You can pull both

linux/amd64

and

arm64

-based* images directly from DockerHub* as follows:

docker pull prefecthq/prefect:2-python3.10 # default: linux/amd64
docker pull prefecthq/prefect:2-python3.10 --platform linux/arm64

And to validate the Prefect version and architecture:

docker run --platform linux/arm64 prefecthq/prefect:2-latest prefect version

Shoutout to @ddelange who led implementation of the feature (#7902). Further enhancements and fixes added in this release: • added

is_schedule_active

to the

Deployment

object • added Kubernetes permissions to the prefect agent YAML template to enable it to query the

kube-system

namespace and retrieve the namespace UUID • improved handling of long-running Kubernetes jobs with the

KubernetesJob

infrastructure block • added support for obscuring secrets in nested block fields in the UI • several documentation fixes and enhancements incl.

KubernetesJob

options, contribution page, keep-alive settings, and Prefect Cloud Quickstart guide. Collection updates: • prefect-openai got a new release

v0.1.1

with a very cool new use case - you can expect a blog post about it soon! • prefect-dask’s landing page has been entirely updated For all enhancements and fixes, check the release notes. P Happy engineering!prefect duck

Hey all - today we announced Premier Consulting Partners. This is a verified, vetted group of integrators and consultants with Prefect expertise. If you need help with a one-off project or a full rebuild of your stack, you can find a partner with the right skills for the job. We have partners with data engineering, data science, ML, and even full stack expertise. Shoutout to our founding 7 partners: • Montreal Analytics • QBiz • APrime • Brooklyn Data • Dataroots • Principal Consulting • Kahan Data Solutions If you'd like to get in touch with one, check out our integrations page on our website. If you think you have what it takes to become certified, reach out to me here on Slack or at chrisr@prefect.io!

We just shipped Prefect 2.7.11 with expanded logging 🪵 One of the best things about Prefect 2 is that not every function to be orchestrated. That flexibility is great, but users told us that even if they weren't orchestrating everything, they still wanted to be able to see logs in a single place. Now, Prefect defaults to displaying a warning instead of raising an error when you use Prefect loggers outside of flow or task runs. We've also added a setting

PREFECT_LOGGING_ORION_WHEN_MISSING_FLOW

to allow configuration of this behavior to silence the warning or raise an error as before. This means that you can attach Prefect's logging handler to existing loggers without breaking your workflows. Check it out:

from prefect import flow
import logging

my_logger = logging.getLogger("my-logger")
<http://my_logger.info|my_logger.info>("outside the flow")

@flow
def foo():
    <http://my_logger.info|my_logger.info>("inside the flow")

if __name__ == "__main__":
    foo()

You can even see messages from

my-logger

in the UI with

PREFECT_LOGGING_EXTRA_LOGGERS

its beautiful:

$ PREFECT_LOGGING_EXTRA_LOGGERS="my-logger" python example.py
example.py:6: UserWarning: Logger 'my-logger' attempted to send logs to Orion without a flow run id. The Orion log handler can only send logs within flow run contexts unless the flow run id is manually provided.
  <http://my_logger.info|my_logger.info>("outside the flow")
18:09:30.518 | INFO    | my-logger - outside the flow
18:09:31.028 | INFO    | prefect.engine - Created flow run 'elated-curassow' for flow 'foo'
18:09:31.104 | INFO    | my-logger - inside the flow
18:09:31.179 | INFO    | Flow run 'elated-curassow' - Finished in state Completed()

Collection updates: • prefect-shell got a new

ShellOperation

block that runs a shell operation • prefect-census got

CensusSync

block that runs a census sync Check out the release notes for information on the other enhancements and fixers in this release. Special thanks to Chia Berry, @Hans Lellelid, and Tomek for their contributions to this release! ⭐ Happy Engineering! marvin

billy mays But wait there's more! Yesterday, we shipped an off-cycle bonus release, 2.7.12. In addition to squashing a few bugs, the release included a highly anticipated requested feature - custom flow and task run names 🎉! A new keyword argument (

flow_run_name

for flows,

task_run_name

for tasks) accepts a string that will be used to create a run name for each run of the function. For example:

from datetime import datetime
from prefect import flow, task

@task(task_run_name="custom-static-name")
def my_task(name):
  print(f"hi {name}")

@flow(flow_run_name="custom-but-fixed-name")
def my_flow(name: str, date: datetime):
  return my_task(name)
  
my_flow()

In order to make these names dynamic, you can template them using the parameter names of the task or flow function, using all of the basic rules of Python string formatting as follows:

from datetime import datetime
from prefect import flow, task

@task(task_run_name="{name}")
def my_task(name):
  print(f"hi {name}")

@flow(flow_run_name="{name}-on-{date:%A}")
def my_flow(name: str, date: datetime):
  return my_task(name)

my_flow()

Check out the Prefect release notes for more information.

Prefect 2.8.0 has been released! 🎆 cloud2 prefect2 prefect duck Prioritize flow runs with work pools With this release, flow runs can now be prioritized among work queues via work pools! Work pools allow you to organize and prioritize work by grouping related work queues together. Within work pools, you can assign a priority to each queue, and flow runs scheduled on higher priority work queues will be run before flow runs scheduled on lower priority work queues. This allows agents to prioritize work that is more important or time-sensitive even if there is a large backlog of flow runs on other work queues in a given work pool. All existing work queues will be assigned to a default work pool named

default-agent-pool

. Creating a new work pool can be done via the Work Pools page in the UI or via the CLI. To create a new work pool named “my-pool” via the CLI:

prefect work-pool create "my-pool"

Each work pool starts out with a default queue. New queues can be added to a work pool via the UI or the CLI. To create a new work queue in a work pool via the CLI:

prefect work-queue create "high-priority" --pool "my-pool"

Deployments can now be assigned to a work queue in a specific work pool. Use the

--pool

flag to specify the work pool and the

--queue

flag to specify the work queue when building a deployment.

prefect deployment build \ 
--pool my-pool \
--queue high-priority \
--name high-priority \
high_priority_flow.py:high_priority_flow

Once a deployment has been created and is scheduling flow runs on a work queue, you can start an agent to pick up those flow runs by starting an agent with the

--pool

flag.

prefect agent start --pool my-pool

Starting an agent with the

--pool

command allows the agent to pick up flow runs for the entire pool even as new queues are added to the pool. If you want to start an agent that only picks up flow runs for a specific queue, you can use the

--queue

flag.

prefect agent start --pool my-pool --queue high-priority

A few other important parts of this release: • The ability to create a flow run from the UI with parameters from a previous run — #8405 • A generic

Webhook

block — #8401 • Override customizations functionality for CLI deployments v— #8349 • It's now possible to reset concurrency limits in CLI to purge existing runs from taking concurrency slots — #8408 To learn more about work pools, and everything else we've added or updated, check out the docs or see the relevant pull requests in the release notes.

Join us live youtube at 1p Eastern today for some hot hot notebooking action with @Kevin Kho, @Matthew Seal from Noteable, and @Will Raphaelson! You'll see how someone working with a notebook can get started quickly with Prefect. Direct link to the stream here. Comments/chat/emoji spam welcome.

Prefect 2.8.1 has been released! 🚀 🦜 New names, same behavior We knew we were onto something big when we first announced Prefect Orion, our second-generation orchestration engine, but we didn't know just how big. Orion's foundational design principles of dynamism, developer experience, and observability have shaped the Prefect 2 codebase to such an extent that it's difficult to tell where Orion ends and other components begin. For example, it's been challenging to communicate clearly about the “Orion API” (the orchestration API), an “Orion Server” (a hosted instance of the API and UI), and individual components of that server. With this release, we've removed references to "Orion" and replaced them with more explicit, conventional nomenclature throughout the codebase. All changes are fully backwards compatible and will follow our standard deprecation cycle of six months. These changes clarify the function of various components, commands, variables, and more. See the deprecated section of the release notes for a full rundown of changes. Note: Many settings have been renamed, but your old settings will be respected. To automatically convert all of the settings in your current profile to the new names, run the

prefect config validate

command. A few other key fixes and enhancements: • We added a

MattermostWebhook

notification block. • You can now pass RRule strings to

--rrule

option in

prefect set-schedule

command. The JSON input it used to require still works, too! • Default deployment parameters now populate correctly in the UI. • We fixed the ability to use anchor dates when setting an interval schedule with the

prefect set-schedule

command. • The docs now have examples of custom automation triggers! Finally, we had two new contributors: • @qheuristics made their first contribution in #8478 • @KernelErr made their first contribution in #8485 See the release notes for a complete list of enhances, fixes, and deprecations in Prefect 2.8.1.

Hi all!

prefect-dbt

v0.3.0 was just released! 🚀 This release includes: • two new blocks:

DbtCloudJob

and

DbtCoreOperation

, making it easier to store configs and trigger+monitor both dbt Cloud jobs and Core CLI commands. •

run_dbt_job_flow

which incorporates the

DbtCloudJob

block. • support for the new

SqlAlchemyConnector

block when using dbt Core with Postgres database. • bug fixes, including saving / loading target config profiles, proper typing of log_format in

GlobalConfigs

• Check out the release notes for more! Check out the new landing page to get started on these new blocks!

Prefect 2.8.2 has been released! 🛠️ This is a quick follow-up to yesterday's release with an important fix and other updates. Prefect 2.8.1 was the week’s big release, so check out its release announcement for a full view of this week’s additions and updates. Here’s what is in 2.8.2: Fixes • Reenable plugin loading in

prefect

module init — #8569 Documentation • Fix logging format override example — #8565 Experimental • Add events client to

PrefectClient

— #8546

Prefect 2.8.3 is here with completion and failure hooks for flows and tasks! 🪝 Completion and Failure Hooks for Flows and Tasks With this release, you can now add client-side hooks that will be called when your flow or task enters a

Completed

or

Failed

state. This is great for any case where you want to execute code without involvement of the Prefect API. Both flows and tasks include

on_completion

and

on_failure

options where a list of callable hooks can be provided. The callable will receive three arguments: •

flow

,

flow_run

, and

state

in the case of a flow hook •

task

,

task_run

, and

state

in the case of a task hook For example, here we add completion hooks to a flow and a task:

from prefect import task, flow

def my_completion_task_hook_1(task, task_run, state):
    print("This is the first hook - Task completed!!!")
    
def my_completion_task_hook_2(task, task_run, state):
  print("This is the second hook - Task completed!!!")
    
def my_completion_flow_hook(flow, flow_run, state):
    print("Flow completed!!!")
    
@task(on_completion=[my_completion_task_hook_1, my_completion_task_hook_2])
def my_task():
    print("This is the task!")

@flow(on_completion=[my_completion_flow_hook])
def my_flow():
    my_task()

if __name__ == "__main__":
    my_flow()

Next, we’ll include a failure hook as well. It’s worth noting that you can supply both

on_completion

and

on_failure

hooks to a flow or task. Only the hooks that are relevant to the final state of the flow or task will be called.

from prefect import task, flow

def my_task_completion_hook(task, task_run, state):
    print("Our task completed successfully!")

def my_task_failure_hook(task, task_run, state):
    print("Our task failed :(")

@task(on_completion=[my_task_completion_hook], on_failure=[my_task_failure_hook])
def my_task():
    raise Exception("Oh no!")

@flow
def my_flow():
    my_task.submit()

if __name__ == "__main__":
    my_flow()

Key enhancements and fixes Other highlights of this week’s enhancements and fixes include: • We added light and dark mode color and contrast enhancements to the UI. #8629 • The

Task.map

type hint now provides type-checker compatibility for async tasks. — #8607 • The Kubernetes Job block now correctly handles timeouts when streaming logs. — #8618 • We fixed date range filter selection on the flow runs UI page. — #8616 See the release notes on GitHub for a full list of this week’s changes! Contributors Thanks to the external contributors to this week’s release! • New contributor @jefflaporte added support for log streaming from multiple containers in a K8s job. • New contributor @AzemaBaptiste added the `prefect-sifflet` collection to our collections catalog. • Veteran contributor @darrida improved the Docker infrastructure block’s handling of Docker version names.

Hi all 👋 We just created the #data-ecosystem channel, a place for talking about new, emerging & exciting technologies. Feel free to post opinions, hot takes, use cases, or your favorite new tool!