https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
announcements
  • a

    Anna Geller

    11/10/2022, 7:39 PM
    Hi everyone! 😛refect2: :cloud2: Prefect 2.6.7 is out and adds: • timeouts for task runs, • colored logging, • obfuscation of secret settings, • dark mode for documentation, • and more! It also includes an incredible amount of usability updates in the UI (e.g., when you navigate to a subflow run, UI will now redirect directly to the logs tab and will additionally display a link to the parent flow run that triggered that child flow run), performance improvements to the scheduler, and lots of bug fixes. We also added a new
    prefect-hightouch
    collection for Hightouch to help you reverse your ETL 😎 As always, you can see the release notes for more details.
    🙌 10
    :cloud2: 13
    :prefect2: 8
    :prefect: 20
    :party-parrot: 12
    🎨 8
    💫 1
  • Prefect Brand Assets
    a

    Anna Geller

    11/15/2022, 2:34 PM
    Hi everyone! 👋 If you want to share public-facing content about 😛refect:Prefect (including blog posts, social media posts, internal slides, and adding Prefect as a logo in your architecture diagrams), we've recently added brand assets that may help. Have a great week!
    :prefect: 12
    📷 10
    :its-beautiful: 7
    👍 5
    🙌 2
    :marvin: 5
    ❤️ 2
    :prefect-duck: 11
    :nod: 2
  • a

    Anna Geller

    11/17/2022, 9:49 PM
    Hi everyone! 👋 We've just released Prefect 2.6.8 😛refect2: :cloud2: This release includes new agent CL options, including the
    --prefetch-seconds
    flag, which is especially useful when interacting with serverless infrastructure. Imagine that you want to schedule a specific flow to run at 9 AM, but it roughly takes a minute to provision a serverless container. With this new CLI flag, your Prefect agent will begin flow run submission earlier than scheduled (in the example below, 60 seconds earlier) to ensure your container starts on time:
    prefect agent start --prefetch-seconds 60 -q default
    The release includes many bug fixes and a new
    prefect-vault
    collection allowing you to interact with Hashicorp Vault - big thanks to Pavel Chekin for contributing it :gratitude-thank-you: As always, you can find more details in the release notes. Happy engineering! 😛refect-duck:
    :cloud2: 10
    :prefect-duck: 12
    🙌 9
    👍 11
    🔥 6
    🙌🏻 1
    :party-parrot: 5
    🙌🏿 1
  • a

    Anna Geller

    11/22/2022, 9:00 PM
    Hi everyone! 👋 Prefect 2.6.9 release has just shipped! 😛refect2::cloud2: It includes several fixes and enhancements to the logging experience. :wood: You will now be able to: • see Dask worker logs in the Prefect UI :dask: • configure logging more easily in tests and CLI commands (a.o. when starting the Orion server) • configure logs to be properly formatted in JSON using
    orjson
    serialization (this is important for logging to external systems such as Datadog - more on that soon!) • avoid duplication when configuring logging e.g. to configure logs to be JSON-formatted, you can now use
    PREFECT_LOGGING_HANDLERS_CONSOLE_FORMATTER="json" python myflow.py
    Apart from that, the
    prefect cloud login
    CLI has been given a serious upgrade. ✨ For a detailed description of that new Cloud CLI login UX, and to see a full list of fixes and enhancements, see the release notes. Happy Engineering! 😛refect:
    🚀 15
    👍 16
    🔑 6
    :dask: 6
    :party-parrot: 16
    ❤️ 4
    💥 11
  • a

    Anna Geller

    12/01/2022, 10:33 PM
    Hi Prefectionists! 😛refect: It's only been a little over a week since the last release, but there are a lot of new features in Prefect 2.7, which was released just now. 😛refect2: :cloud2: This release includes: • Cancellation of flow runs • Pause and resume of flow runs • Logging of print statements • Agent-level flow run concurrency limits • Notification blocks for PagerDuty and Twilio • Improved reporting of crashed infrastructure • Improved handling of failures during HTTP requests • ... and dozens more fixes and enhancements! Thanks a bunch to our five external contributors! 💙 Also, a massive shoutout to @Michael Adkins, who implemented several features of this release, coordinated the entire release work and even wrote most of this announcement and the release post. Check out the release blog post and reach out if you have any questions. Happy Engineering!
    🙌 35
    :blob-attention-gif: 9
    :prefect: 10
    💙 11
    :wizard2: 11
    :party-parrot: 25
    :marvin: 11
  • w

    Will Raphaelson

    12/07/2022, 8:23 PM
    👋 Hi folks - Will here from the Product Team at Prefect! 😛refect:Our mission here at Prefect is to Coordinate the World's Dataflow. As part of this mission, we’re excited to announce the first major milestone of our investment into dataflow observability: Automations. As valued members of our community, I’m excited to invite you all to access Automations in its beta phase. ⚡ Automations in Prefect Cloud enable increased flexibility, control, and responsiveness of your dataflow by allowing you to configure Triggers and Actions in response to events — or lack thereof! -- from across the stack. This forms the foundation of the reactive dataflow that we’ll be rolling out in 2023. 🎏 For our flagship release, we’ve prioritized features to: • Configure Triggers based on Flow Run state changes (or lack thereof) and Work Queue health. • Execute Actions to run/cancel Flow Runs, pause/resume schedules, pause/resume Work Queues, and send beefed up context-aware notifications. 📈 This feature set will be a springboard for a wealth of exciting enhancements to Prefect over the next year, so your thoughts and feedback will be immensely valuable. 😛eople_hugging: If you’re interested in joining the beta, react with your favorite emoji and I’ll add you to the program!
    👋 1
    🥳 1
    :cool-llama: 17
    🚀 47
    :wizard2: 14
    :upvote: 19
    🌶️ 11
    :excellent: 4
    💨 10
    :catjam: 24
    :ahhhhhhhhh: 25
    :blob-attention-gif: 27
    ❤️ 22
    :hattip: 16
    :huzzah: 16
    :you-got-it-dude: 17
    👍 9
    :prefect-duck: 8
    :its-beautiful: 23
    😛anda-dancing: 23
    🙌 20
    :club42: 3
    :race_car: 2
    😮oh: 4
    :oh-yeah: 2
    👀 1
    💙 1
    😒heepy: 14
    💚 3
    🔥 73
    🐛 1
    🎉 50
    :marvin: 35
    ✈️ 32
    🤖 31
    :party-parrot: 51
  • Prefect 2.7.1 Release Announcement
    a

    Anna Geller

    12/08/2022, 10:45 PM
    Hi everyone! 👋 We've just released Prefect 2.7.1 😛refect2: :cloud2: This release adds: • a task run concurrency limits UI page - you can now add new task run concurrency limits and see how many slots are taken at any time • bulk deletion of flows, deployments, and work queues from the UI •
    PREFECT_EXTRA_ENTRYPOINTS
    setting for module injection to allow registration of custom blocks (e.g. during CI/CD) • improved timeout handling on the
    KubernetesJob
    block - this timeout is now disabled by default (i.e.
    job_watch_timeout_seconds
    is set to
    None
    ) • fix for the
    prefect agent start --prefetch-seconds
    command • fix for the flow run URL from notification blocks ...and several more intricate enhancements and fixes you can read about in the release notes. Big thanks to
    t-yuki
    and
    padbk
    for their first contributions. :gratitude-thank-you: Happy engineering! 😛refect-duck:
    🙌 12
    :thank-you: 13
    🚀 16
    ❤️ 9
    💪🏽 1
    🙌🏽 1
    :upvote: 9
    💪 2
  • w

    Will Raphaelson

    12/13/2022, 5:51 PM
    👋 Hi all - Will here again from the Product team. I’m excited to announce that we’re moving Automations into open beta. ⚡ Automations in Prefect Cloud allow for increased flexibility and control of your data stack by allowing you to configure triggers and actions. Using triggers and actions, you can automatically execute critical workflows such as kicking off flow runs, pausing deployments, or sending custom notifications (coming soon!) in response to real-time monitoring events. ❗ Starting today, you’ll see Automations in your workspace sidebar. Give it a try and let us know what you think. 🤖 Happy automating!
    :huzzah: 8
    :prefect: 5
    🙌🏽 1
    ⚡ 7
    :prefect-duck: 6
    🌶️ 5
    💥 3
    :oh-yeah: 9
    :party-parrot: 7
    🤩 5
    🙌 2
    ❤️ 1
    💙 1
    🚀 1
    :gratitude-thank-you: 1
    🤖 10
  • k

    Kalise Richmond

    12/13/2022, 8:26 PM
    Hi Prefect Community! :marvin: We'd like to introduce you to @ask-marvin, our new GPT3-powered finely tuned chat bot where you can ask them any questions in #ask-marvin channel. Marvin is still acquiring knowledge and understanding, despite their advanced intelligence and vast experience. Big shout out to @Serina , @Nate, and @Rob Freedy for their incredible engineering skills building Marvin. Here's a few words from Marvin:
    ⚒️ 3
    :blob-attention-gif: 5
    :marvin: 10
    :marvin-duck: 9
    :prefect-duck: 7
    :marvin-police: 7
    🎉 4
    🤣 4
    :ask-marvin: 18
    j
    • 2
    • 1
  • k

    Kalise Richmond

    12/14/2022, 5:36 PM
    😒lack: The Prefect Slack community has grown by leaps and bounds (now over 23k people!). 🌎 Today we created geographic-based channels so folks could connect with other people nearby and be kept in the loop for community events. We’ve started with locations where we’ve had Prefect Associate Certification courses recently and where we have lots of users. Feel free to join a geo channel if there’s one for your area. 🙂 If there’s isn’t yet a channel for your area, add a comment to the community thread. If someone has already proposed your area in the thread, please give it a 👍. #geo-nyc #geo-bay-area #geo-dc #geo-chicago #geo-boston #geo-london #geo-berlin #geo-texas #geo-seattle #geo-colorado
    🌍 4
    :party-parrot: 4
    🙊 3
    🤩 3
  • Prefect Release 2.7.2 Announcement
    a

    Anna Geller

    12/15/2022, 10:44 PM
    Hi everyone! 👋 Prefect 2.7.2 is here 😛refect2: :cloud2: This release extends the recently added *pause*/*resume* feature with rescheduling of paused flow runs. You can now pass the
    reschedule=True
    argument to the
    pause_flow_run
    utility to gracefully stop the infrastructure process and put the flow run back into a
    Scheduled
    state. This way, you can free up the infrastructure resources instead of keeping the original flow run process running while waiting for approval.
    💡 Note that: this kind of rescheduling pauses works only with flow runs created from deployments and requires results persistence using either the
    persist_result=True
    option on the flow decorator, or by setting it globally:
    prefect config set PREFECT_RESULTS_PERSIST_BY_DEFAULT='True'
    You can now also manually pause a flow run in progress using the new Pause button directly from the flow run UI page (see the screenshot below). Resuming a run can be performed also from the UI, or from code:
    from prefect import resume_flow_run, pause_flow_run
    
    id_ = "pasy_your_flow_run_uuid_here"
    pause_flow_run(id_, timeout=4200, reschedule=True)
    
    # once you're ready to resume (you can do it from the UI, too):
    resume_flow_run(id_)
    Other enhancements and improvements in this release: • Two new Prefect Collections: prefect-kubernetes :kubernetes: and prefect-bitbucket :bucket: • Task Run Concurrency UI page now includes details and shows active task runs currently using allocated concurrency slots (see the second screenshot) • New Opsgenie notifications block - kudos to Julien Lutran for contributing! :gratitude-thank-you: • Fixes to imports and their performance • New
    SecretDict
    block field that allows to obfuscate nested values in a dictionary on blocks such as
    GcpCredentials
    or
    FugueEngine
    • If you have a flow call (e.g.
    myflow()
    ) in your flow script, not protected by an
    if __name__ == "__main__:
    block, any run from that deployment will now work without giving you a warning about that unprotected call • Several documentation improvements incl. new
    with_options
    usage examples, recipe and tutorial references, Automations docs, and more! For all enhancements and fixes, check the release notes - it even includes a Marvin joke. :marvin-duck: Happy engineering! 🎄😛refect-duck:
    🎉 17
    :blob-attention-gif: 7
    :party-parrot: 9
    :prefect: 7
    :thank-you: 7
    🙌 5
    💙 3
    💫 4
    • 1
    • 1
  • c

    Chris Reuter

    12/21/2022, 5:20 PM
    👋 See you all soon! https://prefect-community.slack.com/archives/C036FRC4KMW/p1671572581358209
    :party-parrot: 1
    🙌 6
    :prefect: 6
    :marvin: 7
  • a

    Anna Geller

    12/21/2022, 7:17 PM
    For those who missed it, here is a

    recording of the ▾

    holiday stream. 🎄🎅 We wish you all Happy Holidays and a great start to the New Year! 🎇
    :blob-attention-gif: 3
    🎆 8
    🎁 3
    🎉 5
    :party-parrot: 1
  • j

    Jeff Hale

    12/22/2022, 10:37 PM
    Prefect 2.7.4 has been released! 🚀 Highlights ✨*:* • Improvements to retry delays: multiple delays, exponential backoff, and jitter • Increase in the default thread limit from 40 to 250 • New `prefect-google-sheets` collection - big thanks to community member @Stefano Cascavilla for creating it! Feel free to head on over to the repo and give it a ⭐️ As always, see the release notes for details. Thank you to everyone for your feedback and especially to @devanshdoshi9 and @quassy for their first contributions! Happy holidays! ☃️
    :marvin: 8
    🙌 9
    :gratitude-thank-you: 7
    🎉 2
    🎁 3
    ☃️ 5
    🚀 11
    :prefect: 7
  • j

    Jeff Hale

    12/29/2022, 11:41 PM
    Prefect 2.7.5 has been released! 🎉 Highlights • Schedule flow runs from the CLI with
    --start-in
    or
    --start-at
    e.g.
    prefect deployment run foo/test --start-at "3pm tomorrow"
    • Get the logs for a flow run using
    prefect flow-run logs <flow run UUID>
    e.g.
    prefect flow-run logs 7aec7a60-a0ab-4f3e-9f2a-479cd85a2aaf
    Collection updates • Added
    BigQueryWarehouse
    block in
    prefect-gcp
    v0.2.1 • Added
    AirbyteConnection
    block in
    prefect-airbyte
    v0.2.0 • Added dbt Cloud metadata API client to
    DbtCloudCredentials
    in
    prefect-dbt
    v0.2.7 As always, see the release notes for details. Thank you to everyone for your feedback and especially to @ohadch and @mohitsaxenaknoldus for their first contributions! Happy New Year! 🥳
    🚀 8
    🔥 7
    ❤️ 6
    :prefect: 5
    :gratitude-thank-you: 5
    👍 2
    🎆 4
    🎉 13
    👍🏽 1
    🎈 1
    🙌🏽 1
    🙌 7
  • Announcement for Cloud 1 users with auth tokens before May 2020
    k

    Kingsley Blatter

    01/05/2023, 10:01 PM
    Announcement only for Cloud 1.0 users that created auth tokens before May 2020: Last night (Wednesday, January 4th) we were notified by CircleCI of a security alert. The security alert does not indicate that Prefect was impacted, but strongly recommends that all CircleCI users rotate their secrets. As a consequence of this rotation, we will no longer be able to verify auth tokens that were generated prior to approximately May 2020. Therefore, Prefect Agents that use legacy “Runner” or “Tenant” tokens generated before May 2020 will stop working and will need to be provided with new API keys to continue running flows. Users who adopted Prefect API Tokens or API Keys after May 2020 should not be affected.
    :thank-you: 7
    💙 5
    :marvin-police: 4
    :prefect: 4
    👏 4
  • Prefect 2.7.7 Release
    a

    Anna Geller

    01/06/2023, 11:03 PM
    Hi everyone, happy Friday! 👋 Prefect 2.7.7 is here 😛refect2: :cloud2: The API reference documentation has been completely rehauled with improved navigation and samples. You can see the REST API documentation on Prefect Cloud, as well as rendered in our open-source documentation. We've also improved the parsing and rendering of reference documentation for our Python API. See the tabular @flow decorator docs as an example. This release also includes lots of enhancements related to blocks: • PR 7862 added
    validate
    flag on the
    Block.load
    method to facilitate block document migration, and PR 8056 allowed schema migration of block documents using
    Block.save
    • when you register new block types with the
    prefect block register
    CLI command, you will now get a link to the UI Blocks page to easily create new blocks (8017) • when you try to save a block with an invalid name, you'll see better error messages (8038) Apart from that, there are improvements and fixes to (among others!): • flow timeouts (7993) •
    prefect cloud login
    (8034) • Kubernetes job (8018) and its documentation (8044) •
    SecretDict
    serialization (8074) • annotations (7263) and their serialization (8037) • task run cache key size (7275) • RRule schedule length - there is a limit now (7762) • deployment troubleshooting - we added a log message when Prefect downloads flow code from storage and from which storage location (8075) • the result factory, which now avoids creating unnecessary DB connections (8072) • documentation - lots of improvements here, including not only docs but also a new parser and renderer (7855) For all enhancements and fixes, check the release notes. Happy engineering and have a great weekend!😛refect-duck:
    :prefect: 7
    :anna: 1
    :prefect-duck: 6
    :thank-you: 4
    :its-beautiful: 3
    :upvote: 7
    🔥 22
    :marvin-duck: 12
    🚀 14
    ⭐ 8
  • w

    Will Raphaelson

    01/18/2023, 8:13 PM
    👋 Hi folks - Will here from the Product Team at Prefect! About a month ago, I announced the beta release of our first major dataflow observability feature, Automations. Today, I’m excited to promote that feature to GA across all Cloud workspaces. ⚡ Automations in Prefect Cloud enable increased flexibility, control, and responsiveness of your dataflow by allowing you to configure Triggers and Actions in response to events — or lack thereof! -- from across the stack. This forms the foundation of the reactive dataflow capabilities that we’ll be rolling out in 2023. 🎏 For our flagship release, we’ve prioritized features to: • Configure Triggers based on Flow Run state changes, Work Queue health, or custom criteria via the API. • Execute Actions to run/cancel Flow Runs, pause/resume schedules, pause/resume Work Queues, and send beefed up context-aware notifications with support for Jinja templating. 📈 This feature set will be a springboard for a wealth of exciting enhancements to Prefect over the next year, so as always, let us know what you think. 😛eople_hugging: Happy automating! -WR
    😒quirtle_cool: 3
    🎉 1
    🙌 3
    🧑‍🚀 1
    💨 2
    💯 4
    😒nap-point: 4
    :cloud2: 3
    :prefect: 3
    🆒 1
    :meow_code: 6
    :all-coming-together: 4
    🛺 9
    :wizard2: 9
    👏 6
  • a

    Anna Geller

    01/19/2023, 10:57 PM
    Prefect 2.7.9 is here! 😛refect2: :cloud2: This release ships with multi-architecture development Docker images! :docker_ship: This means that you can now pull both
    linux/amd64
    and
    arm64
    -based development images directly from DockerHub, for example:
    docker pull prefecthq/prefect-dev:main-python3.10 # default: linux/amd64
    docker pull prefecthq/prefect-dev:main-python3.10 --platform linux/arm64
    There is also a new Prefect OpenAI collection for all AI fans! We've also: • :github: added a new interface to filter
    .git
    files when using the
    GitHub
    block • :terminal: added the
    --head
    and
    --num-logs
    flags to the
    flow-run logs
    CLI: ◦
    prefect flow-run logs run_uuid --head
    will give you 20 log messages for a given flow run UUID ◦
    prefect flow-run logs run_uuid --num-logs 5
    will give you 5 log messages for a given flow run UUID • 🐛 fixed several bugs, a.o. one for blocks and one for Windows subprocesses • 📚 added a new recipe contribution documentation page • :aws: added a new AWS Chalice recipe Big thanks to ddelange for consistently working on contributing several PRs to add the
    arm64
    -based Docker images as well as to
    mj0nez
    , and
    ohadch
    for contributing to Prefect!😛refect: For all enhancements and fixes, check the release notes. Happy engineering!😛refect-duck:
    :prefect: 12
    :prefect2: 8
    :prefect-duck: 8
    🤯 3
    :its-beautiful: 4
    :highfive: 4
    :cloud2: 9
    🎉 11
    :docker_ship: 2
  • c

    Chris Reuter

    01/20/2023, 7:34 PM
    Good Afternoon & hi everyone! Today we are announcing that legacy Prefect Cloud 1 accounts will be frozen for Free tier users on May 1st 2023, and for Starter & Standard tier users on monthly plans on July 15th 2023. This means that your account and tenant will become read-only and your flows & tasks will no longer run. Prefect 2 has seen exciting development advancements over the past year, and Prefect Cloud 2 is available with a free tier and a simple pricing model based on users & workspaces. To help our remaining users as they transition, we are providing upgrade support to all users via written guides, videos and help from real Prefect engineers. To get upgrade support, visit https://upgrade.prefect.io/. Note that this freeze only pertains to Free, Starter and Standard users of Prefect Cloud 1 on monthly plans, available at cloud.prefect.io. Enterprise users, those with annual agreements and accounts on Prefect Cloud 2 are not impacted by this announcement.
    :gratitude-thank-you: 12
    😒aluting_face: 15
    📣 6
    :highfive: 3
    :prefect: 6
    :cloud2: 7
    💙 8
    😮ld-man-yells-at-cloud: 7
    ☁️ 4
    😬 3
    2️⃣ 1
    g
    • 2
    • 1
  • a

    Anna Geller

    01/26/2023, 10:56 PM
    Hi everyone! Prefect 2.7.10 has been released! 😛refect2: :cloud2: We're excited to announce an upgrade to our flow run cancellation feature, resolving many common issues. When cancellation is requested, the agent now sends a termination signal (SIGTERM) to the flow run infrastructure. Previously, this signal resulted in the immediate exit of the flow run. Now, we detect the signal and attempt to shut down the process gracefully. This gives the run an opportunity to clean up any resources it is managing. If the flow run does not gracefully exit in a reasonable time, it will be killed. More details about cancellation improvements: 1. We improved how we handle runs in the process of cancelling -- when a run is cancelled, it's first placed in a "cancelling" state, then moved to a "cancelled" state when cancellation is complete. 2. Previously, concurrency slots were released as soon as cancellation was requested. Now, the flow run will continue to occupy concurrency slots until a "cancelled" state is reached. 3. We added cleanup of tasks and subflows belonging to cancelled flow runs. Previously, these tasks and subflows could be left in a "running" state. This can cause problems with concurrency slot consumption and restarts, so we've added a service that updates the states of the children of recently cancelled flow runs. Additionally, from this release on, we'll be publishing multi-architecture Docker images to DockerHub not only for development images, but also for production images! :docker_ship: You can pull both
    linux/amd64
    and
    arm64
    -based* images directly from DockerHub* as follows:
    docker pull prefecthq/prefect:2-python3.10 # default: linux/amd64
    docker pull prefecthq/prefect:2-python3.10 --platform linux/arm64
    And to validate the Prefect version and architecture:
    docker run --platform linux/arm64 prefecthq/prefect:2-latest prefect version
    Shoutout to @ddelange who led implementation of the feature (#7902). Further enhancements and fixes added in this release: • added
    is_schedule_active
    to the
    Deployment
    object • added Kubernetes permissions to the prefect agent YAML template to enable it to query the
    kube-system
    namespace and retrieve the namespace UUID • improved handling of long-running Kubernetes jobs with the
    KubernetesJob
    infrastructure block • added support for obscuring secrets in nested block fields in the UI • several documentation fixes and enhancements incl.
    KubernetesJob
    options, contribution page, keep-alive settings, and Prefect Cloud Quickstart guide. Collection updates: • prefect-openai got a new release
    v0.1.1
    with a very cool new use case - you can expect a blog post about it soon! • prefect-dask’s landing page has been entirely updated For all enhancements and fixes, check the release notes. 😛refect: Happy engineering!😛refect-duck:
    🎉 23
    :blob-attention-gif: 13
    :prefect: 9
    :cloud2: 7
    🙌 8
    🥳 2
    🙌🏻 1
  • c

    Chris Reuter

    01/31/2023, 4:43 PM
    Hey all - today we announced Premier Consulting Partners. This is a verified, vetted group of integrators and consultants with Prefect expertise. If you need help with a one-off project or a full rebuild of your stack, you can find a partner with the right skills for the job. We have partners with data engineering, data science, ML, and even full stack expertise. Shoutout to our founding 7 partners: • Montreal Analytics • QBiz • APrime • Brooklyn Data • Dataroots • Principal Consulting • Kahan Data Solutions If you'd like to get in touch with one, check out our integrations page on our website. If you think you have what it takes to become certified, reach out to me here on Slack or at chrisr@prefect.io!
    🤝 19
    🙌 9
    💙 13
    👍 7
    :its-beautiful: 6
  • j

    justabill

    02/03/2023, 1:17 AM
    We just shipped Prefect 2.7.11 with expanded logging :wood: One of the best things about Prefect 2 is that not every function to be orchestrated. That flexibility is great, but users told us that even if they weren't orchestrating everything, they still wanted to be able to see logs in a single place. Now, Prefect defaults to displaying a warning instead of raising an error when you use Prefect loggers outside of flow or task runs. We've also added a setting
    PREFECT_LOGGING_ORION_WHEN_MISSING_FLOW
    to allow configuration of this behavior to silence the warning or raise an error as before. This means that you can attach Prefect's logging handler to existing loggers without breaking your workflows. Check it out:
    from prefect import flow
    import logging
    
    my_logger = logging.getLogger("my-logger")
    <http://my_logger.info|my_logger.info>("outside the flow")
    
    @flow
    def foo():
        <http://my_logger.info|my_logger.info>("inside the flow")
    
    if __name__ == "__main__":
        foo()
    You can even see messages from
    my-logger
    in the UI with
    PREFECT_LOGGING_EXTRA_LOGGERS
    :its-beautiful::
    $ PREFECT_LOGGING_EXTRA_LOGGERS="my-logger" python example.py
    example.py:6: UserWarning: Logger 'my-logger' attempted to send logs to Orion without a flow run id. The Orion log handler can only send logs within flow run contexts unless the flow run id is manually provided.
      <http://my_logger.info|my_logger.info>("outside the flow")
    18:09:30.518 | INFO    | my-logger - outside the flow
    18:09:31.028 | INFO    | prefect.engine - Created flow run 'elated-curassow' for flow 'foo'
    18:09:31.104 | INFO    | my-logger - inside the flow
    18:09:31.179 | INFO    | Flow run 'elated-curassow' - Finished in state Completed()
    Collection updates: • prefect-shell got a new
    ShellOperation
    block that runs a shell operation • prefect-census got
    CensusSync
    block that runs a census sync Check out the release notes for information on the other enhancements and fixers in this release. Special thanks to Chia Berry, @Hans Lellelid, and Tomek for their contributions to this release! ⭐ Happy Engineering! :marvin:
    :marvin: 20
    :party-parrot: 11
    :gratitude-thank-you: 8
    🥳 9
    👍 6
    🙌 5
    :blob-attention-gif: 4
    😒quirtle_cool: 2
  • j

    justabill

    02/07/2023, 2:00 PM
    :billy-mays: But wait there's more! Yesterday, we shipped an off-cycle bonus release, 2.7.12. In addition to squashing a few bugs, the release included a highly anticipated requested feature - custom flow and task run names 😮h-yeah:! A new keyword argument (
    flow_run_name
    for flows,
    task_run_name
    for tasks) accepts a string that will be used to create a run name for each run of the function. For example:
    from datetime import datetime
    from prefect import flow, task
    
    @task(task_run_name="custom-static-name")
    def my_task(name):
      print(f"hi {name}")
    
    @flow(flow_run_name="custom-but-fixed-name")
    def my_flow(name: str, date: datetime):
      return my_task(name)
      
    my_flow()
    In order to make these names dynamic, you can template them using the parameter names of the task or flow function, using all of the basic rules of Python string formatting as follows:
    from datetime import datetime
    from prefect import flow, task
    
    @task(task_run_name="{name}")
    def my_task(name):
      print(f"hi {name}")
    
    @flow(flow_run_name="{name}-on-{date:%A}")
    def my_flow(name: str, date: datetime):
      return my_task(name)
    
    my_flow()
    Check out the Prefect release notes for more information.
    🙌 9
    🔥 22
    👍 10
    🎉 7
    :prefect: 8
    ✨ 7
    :gratitude-thank-you: 8
    📣 7
    :oh-yeah: 7
    🚀 3
    🥳 7
    😮oh: 5
    :party-parrot: 1
    👍🏻 1
  • r

    Ryan Peden

    02/09/2023, 10:06 PM
    Prefect 2.8.0 has been released! 🎆 :cloud2: 😛refect2: 😛refect-duck: Prioritize flow runs with work pools With this release, flow runs can now be prioritized among work queues via work pools! Work pools allow you to organize and prioritize work by grouping related work queues together. Within work pools, you can assign a priority to each queue, and flow runs scheduled on higher priority work queues will be run before flow runs scheduled on lower priority work queues. This allows agents to prioritize work that is more important or time-sensitive even if there is a large backlog of flow runs on other work queues in a given work pool. All existing work queues will be assigned to a default work pool named
    default-agent-pool
    . Creating a new work pool can be done via the Work Pools page in the UI or via the CLI. To create a new work pool named “my-pool” via the CLI:
    prefect work-pool create "my-pool"
    Each work pool starts out with a default queue. New queues can be added to a work pool via the UI or the CLI. To create a new work queue in a work pool via the CLI:
    prefect work-queue create "high-priority" --pool "my-pool"
    Deployments can now be assigned to a work queue in a specific work pool. Use the
    --pool
    flag to specify the work pool and the
    --queue
    flag to specify the work queue when building a deployment.
    prefect deployment build \ 
    --pool my-pool \
    --queue high-priority \
    --name high-priority \
    high_priority_flow.py:high_priority_flow
    Once a deployment has been created and is scheduling flow runs on a work queue, you can start an agent to pick up those flow runs by starting an agent with the
    --pool
    flag.
    prefect agent start --pool my-pool
    Starting an agent with the
    --pool
    command allows the agent to pick up flow runs for the entire pool even as new queues are added to the pool. If you want to start an agent that only picks up flow runs for a specific queue, you can use the
    --queue
    flag.
    prefect agent start --pool my-pool --queue high-priority
    A few other important parts of this release: • The ability to create a flow run from the UI with parameters from a previous run — #8405 • A generic
    Webhook
    block — #8401 • Override customizations functionality for CLI deployments v— #8349 • It's now possible to reset concurrency limits in CLI to purge existing runs from taking concurrency slots — #8408 To learn more about work pools, and everything else we've added or updated, check out the docs or see the relevant pull requests in the release notes.
    :party-parrot: 2
    🙌 13
    :marvin: 10
    :blob-attention-gif: 3
    :oh-yeah: 7
    🔥 6
    🎉 8
  • c

    Chris Reuter

    02/16/2023, 3:19 PM
    Join us live :youtube: at 1p Eastern today for some hot hot notebooking action with @Kevin Kho, @Matthew Seal from Noteable, and @Will Raphaelson! You'll see how someone working with a notebook can get started quickly with Prefect. Direct link to the stream here. Comments/chat/emoji spam welcome.
    :marvin: 5
    :blob-attention-gif: 2
    🙌 2
  • r

    Ryan Peden

    02/16/2023, 7:12 PM
    Prefect 2.8.1 has been released! 🚀 😛arty-parrot: New names, same behavior We knew we were onto something big when we first announced Prefect Orion, our second-generation orchestration engine, but we didn't know just how big. Orion's foundational design principles of dynamism, developer experience, and observability have shaped the Prefect 2 codebase to such an extent that it's difficult to tell where Orion ends and other components begin. For example, it's been challenging to communicate clearly about the “Orion API” (the orchestration API), an “Orion Server” (a hosted instance of the API and UI), and individual components of that server. With this release, we've removed references to "Orion" and replaced them with more explicit, conventional nomenclature throughout the codebase. All changes are fully backwards compatible and will follow our standard deprecation cycle of six months. These changes clarify the function of various components, commands, variables, and more. See the deprecated section of the release notes for a full rundown of changes. Note: Many settings have been renamed, but your old settings will be respected. To automatically convert all of the settings in your current profile to the new names, run the
    prefect config validate
    command. A few other key fixes and enhancements: • We added a
    MattermostWebhook
    notification block. • You can now pass RRule strings to
    --rrule
    option in
    prefect set-schedule
    command. The JSON input it used to require still works, too! • Default deployment parameters now populate correctly in the UI. • We fixed the ability to use anchor dates when setting an interval schedule with the
    prefect set-schedule
    command. • The docs now have examples of custom automation triggers! Finally, we had two new contributors: • @qheuristics made their first contribution in #8478 • @KernelErr made their first contribution in #8485 See the release notes for a complete list of enhances, fixes, and deprecations in Prefect 2.8.1.
    🙌 13
    :oh-yeah: 3
    :typingcat: 6
    :marvin: 7
    :prefect: 2
  • k

    Kalise Richmond

    02/16/2023, 8:13 PM
    Hi all!
    prefect-dbt
    v0.3.0 was just released! 🚀 This release includes: • two new blocks:
    DbtCloudJob
    and
    DbtCoreOperation
    , making it easier to store configs and trigger+monitor both dbt Cloud jobs and Core CLI commands. •
    run_dbt_job_flow
    which incorporates the
    DbtCloudJob
    block. • support for the new
    SqlAlchemyConnector
    block when using dbt Core with Postgres database. • bug fixes, including saving / loading target config profiles, proper typing of log_format in
    GlobalConfigs
    • Check out the release notes for more! Check out the new landing page to get started on these new blocks!
    👀 5
    💥 7
    👏 6
    :dbt: 11
  • r

    Ryan Peden

    02/17/2023, 11:18 PM
    Prefect 2.8.2 has been released! 🛠️ This is a quick follow-up to yesterday's release with an important fix and other updates. Prefect 2.8.1 was the week’s big release, so check out its release announcement for a full view of this week’s additions and updates. Here’s what is in 2.8.2: Fixes • Reenable plugin loading in
    prefect
    module init — #8569 Documentation • Fix logging format override example — #8565 Experimental • Add events client to
    PrefectClient
    — #8546
    🎉 10
    🙌 5
    👍 6
    :prefect: 6
    :prefect-duck: 5
  • r

    Ryan Peden

    02/24/2023, 1:01 AM
    Prefect 2.8.3 is here with completion and failure hooks for flows and tasks! :hook: Completion and Failure Hooks for Flows and Tasks With this release, you can now add client-side hooks that will be called when your flow or task enters a
    Completed
    or
    Failed
    state. This is great for any case where you want to execute code without involvement of the Prefect API. Both flows and tasks include
    on_completion
    and
    on_failure
    options where a list of callable hooks can be provided. The callable will receive three arguments: •
    flow
    ,
    flow_run
    , and
    state
    in the case of a flow hook •
    task
    ,
    task_run
    , and
    state
    in the case of a task hook For example, here we add completion hooks to a flow and a task:
    from prefect import task, flow
    
    def my_completion_task_hook_1(task, task_run, state):
        print("This is the first hook - Task completed!!!")
        
    def my_completion_task_hook_2(task, task_run, state):
      print("This is the second hook - Task completed!!!")
        
    def my_completion_flow_hook(flow, flow_run, state):
        print("Flow completed!!!")
        
    @task(on_completion=[my_completion_task_hook_1, my_completion_task_hook_2])
    def my_task():
        print("This is the task!")
    
    @flow(on_completion=[my_completion_flow_hook])
    def my_flow():
        my_task()
    
    if __name__ == "__main__":
        my_flow()
    Next, we’ll include a failure hook as well. It’s worth noting that you can supply both
    on_completion
    and
    on_failure
    hooks to a flow or task. Only the hooks that are relevant to the final state of the flow or task will be called.
    from prefect import task, flow
    
    def my_task_completion_hook(task, task_run, state):
        print("Our task completed successfully!")
    
    def my_task_failure_hook(task, task_run, state):
        print("Our task failed :(")
    
    @task(on_completion=[my_task_completion_hook], on_failure=[my_task_failure_hook])
    def my_task():
        raise Exception("Oh no!")
    
    @flow
    def my_flow():
        my_task.submit()
    
    if __name__ == "__main__":
        my_flow()
    Key enhancements and fixes Other highlights of this week’s enhancements and fixes include: • We added light and dark mode color and contrast enhancements to the UI. #8629 • The
    Task.map
    type hint now provides type-checker compatibility for async tasks. — #8607 • The Kubernetes Job block now correctly handles timeouts when streaming logs. — #8618 • We fixed date range filter selection on the flow runs UI page. — #8616 See the release notes on GitHub for a full list of this week’s changes! Contributors Thanks to the external contributors to this week’s release! • New contributor @jefflaporte added support for log streaming from multiple containers in a K8s job. • New contributor @AzemaBaptiste added the `prefect-sifflet` collection to our collections catalog. • Veteran contributor @darrida improved the Docker infrastructure block’s handling of Docker version names.
    🎉 8
    :party-parrot: 6
    😒lick: 4
    🙌 8
    :thank-you: 4
Powered by Linen
Title
r

Ryan Peden

02/24/2023, 1:01 AM
Prefect 2.8.3 is here with completion and failure hooks for flows and tasks! :hook: Completion and Failure Hooks for Flows and Tasks With this release, you can now add client-side hooks that will be called when your flow or task enters a
Completed
or
Failed
state. This is great for any case where you want to execute code without involvement of the Prefect API. Both flows and tasks include
on_completion
and
on_failure
options where a list of callable hooks can be provided. The callable will receive three arguments: •
flow
,
flow_run
, and
state
in the case of a flow hook •
task
,
task_run
, and
state
in the case of a task hook For example, here we add completion hooks to a flow and a task:
from prefect import task, flow

def my_completion_task_hook_1(task, task_run, state):
    print("This is the first hook - Task completed!!!")
    
def my_completion_task_hook_2(task, task_run, state):
  print("This is the second hook - Task completed!!!")
    
def my_completion_flow_hook(flow, flow_run, state):
    print("Flow completed!!!")
    
@task(on_completion=[my_completion_task_hook_1, my_completion_task_hook_2])
def my_task():
    print("This is the task!")

@flow(on_completion=[my_completion_flow_hook])
def my_flow():
    my_task()

if __name__ == "__main__":
    my_flow()
Next, we’ll include a failure hook as well. It’s worth noting that you can supply both
on_completion
and
on_failure
hooks to a flow or task. Only the hooks that are relevant to the final state of the flow or task will be called.
from prefect import task, flow

def my_task_completion_hook(task, task_run, state):
    print("Our task completed successfully!")

def my_task_failure_hook(task, task_run, state):
    print("Our task failed :(")

@task(on_completion=[my_task_completion_hook], on_failure=[my_task_failure_hook])
def my_task():
    raise Exception("Oh no!")

@flow
def my_flow():
    my_task.submit()

if __name__ == "__main__":
    my_flow()
Key enhancements and fixes Other highlights of this week’s enhancements and fixes include: • We added light and dark mode color and contrast enhancements to the UI. #8629 • The
Task.map
type hint now provides type-checker compatibility for async tasks. — #8607 • The Kubernetes Job block now correctly handles timeouts when streaming logs. — #8618 • We fixed date range filter selection on the flow runs UI page. — #8616 See the release notes on GitHub for a full list of this week’s changes! Contributors Thanks to the external contributors to this week’s release! • New contributor @jefflaporte added support for log streaming from multiple containers in a K8s job. • New contributor @AzemaBaptiste added the `prefect-sifflet` collection to our collections catalog. • Veteran contributor @darrida improved the Docker infrastructure block’s handling of Docker version names.
🎉 8
:party-parrot: 6
😒lick: 4
🙌 8
:thank-you: 4
View count: 8