Anna Geller
07/01/2022, 2:22 PMjustabill
07/05/2022, 6:34 PM2.0b8
release will include a breaking change to the interface between Prefect Cloud 2.0 and your agents. You will need to upgrade your agents to ensure that your deployments continue to run as scheduled.
We will announce both when the release is available and when Cloud 2.0 has been updated to work with the new client 📣.
This release will not impact Cloud 1.0. <!here>Anna Geller
07/06/2022, 12:40 AMAnna Geller
07/08/2022, 4:07 PM2.0b8
with some really cool features - @justabillwill tell you more about it soon.
You can take it for a spin already:
pip install -U "prefect>=2.0b8"
justabill
07/08/2022, 4:45 PM2.0b8
! It's full of exciting new features and refinements to existing concepts, including:
Flow Run Retries 🔄
Flow run retries have been one of our most requested features, especially given how easy it is to run a flow as a "subflow" or "child flow" with Prefect 2.0. Flow run retries are configured just as task retries are - with the retries
and retry_delay_seconds
parameters.
If both a task and its flow have retries configured, tasks within the flow will retry up to their specified task retry limit for each flow run. For example, if you have a flow configured with a limit of 2 retries (up to 3 total runs, including the initial attempt), and a task in the flow configured with 3 retries (up to 4 attempts per flow run, including the initial attempt). The task could run up to a total of 12 attempts, since task retry limits are reset after each flow run or flow run attempt.
Notifications 📣
You can always visit the Prefect UI to get a comprehensive view of the state of all of your flows, but when something goes wrong with one of them, you need that information immediately. Notifications can alert you and your team when any flow enters any state you specify, with or without specific tags.
To create a notification, go to the new Notifications page via the sidebar navigation and select “Create Notification.” Notifications are structured just as you would describe them to someone. For example, if I want to get a Slack message every time my daily-ETL flow fails, my notification will simply read:
If a run of any flow with any tag enters a failed state, send a notification to SlackCurrently, notifications can only be sent to a Slack webhook (or email addresses if you are using Prefect Cloud 2.0). Over time, notifications will support additional messaging services. Let us know which messaging services you’d like to send your notifications to! Flow packaging and deployment 📦 We've revisited our flow packaging and deployment UX, making it both more powerful and easier to use. `DeploymentSpec`s are now just `Deployment`s. Most of the fields are unchanged, but there are a few differences: • The
flow_storage
field has been replaced with a packager
field.
• The flow_location
, flow_name
, and flow
parameters are now just flow
.
You can deploy a flow directly, or you can package it with the new packaging
module. Packaging enables you to bundle a flow with its dependencies, including Python packages. Packaging produces a Package, the artifact that contains your flow and its dependencies, as well as a Manifest, a JSON description of the package and what is inside of it.
There are several ways to create a package, with more to come. For instance, you can package your flow as a Docker image containing the flow and the runtime environment necessary to run it and register it with Docker. Alternatively, you could package your flow as a reference to the import path of a package stored directly in the Prefect Orion database. Learn more in the Deployment concept documentation.
You can continue to use your existing `DeploymentSpec`s, but they are deprecated and will be removed in the coming weeks.
Blocks 🧱
We've been working on Blocks behind the scenes for a while. Whether you know it or not, if you've used the past few releases, you've used them. Blocks enable you to securely store configuration with the Prefect Orion server and access it from your code later with just a simple reference. Think of Blocks as secure, UI-editable, type-checked environment variables. We're starting with just a few Blocks - mostly storage, but over time we’ll expand this pattern to include every tool and service in the growing modern data stack. You'll be able to set up access to your entire stack once in just a few minutes, then manage access forever without editing your code.
Breaking Changes 🚨
This release removes the deprecated DaskTaskRunner
:dask: and RayTaskRunner
:ray: from the core library, breaking existing references to them. You can find them in their respective collections prefect-ray and prefect-dask. It also removes the previously deprecated restart policy for the KubernetesFlowRunnner
. Most importantly, there are new breaking changes to the Deployments interface described below.
Special thanks to @Cole Murray, @Oliver Mannion, @Steve Flitcroft, and Laerte Pereira for their contributions to this release!Anna Geller
07/08/2022, 6:08 PMKevin Kho
07/08/2022, 6:46 PMChris Reuter
07/18/2022, 6:11 PMjustabill
07/19/2022, 10:47 PM2.0b9
, includes breaking changes and deprecations in preparation for Prefect 2 graduating from its beta period to General Availability.
With next week’s release on July 27th, Prefect 2.0 will become the default package installed when you execute pip install prefect
. Please ensure that your package management process enables you to make the transition to Prefect 2.0 when the time is right for you.
🚨 Note: Flows written with Prefect 1.0 will require modifications to run with Prefect 2.0
Code as workflows :terminal:
As Prefect 2 usage has grown, we’ve observed a pattern among users, especially folks that were not previously users of Prefect 1. Working with Prefect was so much like working in native Python, users were often surprised that their tasks returned futures and states, Prefect objects, rather than results, the data that their Python functions were handling. This led to unfamiliar, potentially intimidating, errors in some cases.
With this release, Prefect moves one step closer to code as workflows - tasks now return the results of their functions, rather than their states, by default. This means that you can truly take most native Python scripts, add the relevant @flow and @task decorators, and start running that script as a flow, benefitting from the observability and resilience that Prefect provides.
States and futures are still important concepts in dictating and understanding the behavior of flows. You will still be able to easily access and use them with the .submit()
method. You will need to modify tasks in existing Prefect 2 flows to use this method to continue working as before.
Other improvements and bug fixes :level-up:
• A new Secret
block can store a string that is encrypted at rest_* as well as obfuscated in logs and the UI
• Date filters on the flow run page in the UI now support filtering by date and time
• Each work queue page in the UI now includes a command to start a corresponding agent
• Tutorials have been updated for increased clarity and consistency
• Cron schedule setting errors are now more informative
• Prefect now still works even if the active profile is missing
• Conda requirements regex now supports underscores and dots
• The previously deprecated DeploymentSpec
has been removed
As always, for detailed release notes with links to the relevant PRs, see release notes on GitHub.
_* While the string itself is encrypted at rest, the encryption key is accessible via the database, please take this into consideration for security planning purposesAnna Geller
07/22/2022, 12:04 AMprefect.infrastructure
module allowing you to easily deploy infrastructure for running docker containers, subprocesses, and Kubernetes jobs.
Please don't worry about missing release notes - we'll provide a detailed description of the changes with the General Availability release next week.
If you currently use Prefect 1.0, please make sure you've pinned your Prefect version explicitly (including your Docker images!) to avoid any surprises, as Prefect 2.0 will become the default version when installing Prefect starting next Wednesday.
If something isn't working or you are confused, please send us a Slack message or bear with us until Wednesday.
We appreciate your patience, and we promise it will be worth it! 🚀Anna Geller
07/22/2022, 12:07 PMpip install prefect==1.2.4
🔴 Installing Prefect with an implicit version will install Prefect 2.0, which will break your existing flows if they have been written using Prefect 1.0:
pip install prefect
---
#2 When working with Docker images in your run configuration or custom Dockerfiles, use an explicit tag indicating the right Prefect version you need:
:large_green_circle: This is the most recent Prefect 1.0 Docker image tag for Python 3.9, which is 1.2.4 as of today:
prefecthq/prefect:1.2.4-python3.9
🔴 Using an implicit, i.e., the latest Docker image tag for Python 3.9 (or any other Python version) will leverage Prefect 2.0 as the default:
prefecthq/prefect:latest-python3.9
⚠️ This means that the latest tags of Docker images on your Prefect 1.0 run configurations or custom Dockerfiles will be using Prefect 2.0, which will break your flows unless you make that change of explicitly pinning your dependencies! Please make that change and reregister your flows if needed as soon as possible to avoid any surprises.
If you have any questions about it, please reach out.Chris Reuter
07/22/2022, 7:00 PMChris Reuter
07/25/2022, 2:59 PMjustabill
07/27/2022, 9:10 PMChris Reuter
07/28/2022, 2:07 PMAnna Geller
08/02/2022, 5:57 PMAnna Geller
08/02/2022, 10:58 PMprefect deployment build
, you might want to skip the upload of some files which are relevant for local development but not intended for use in production. To solve that problem, we introduced the .prefectignore
file! 😎
How to use .prefectignore
?
Add the file .prefectignore
to your root project directory and follow the same syntax as in .gitignore
to make Prefect aware that those files and directories should be ignored at deployment build time.
• The syntax supports pattern matching, so an entry of *.pyc
will ensure all .pyc files are ignored.
• Prefect will add a default .prefectignore
file if you don’t add it yourself - you can customize it at any time.
---
Azure file system block
If you are an Azure user, you can start using that as a storage of your flow code and custom modules. Simply add an Azure block in your UI or suing the following Python syntax:
from prefect.filesystems import Azure
block = Azure(azure_storage_connection_string="paste_the_string_here")
block.save("dev")
And reference that block in your deployment CLI:
prefect deployment build flow.py:flowname \
--name deploy_name --tag dev -sb azure/dev
---
Issue templates
You can now submit bug reports and feature enhancements using our improved issue templates. Shoutout to @Peyton Runyan for being so proactive and streamlining that experience for everyone!
---
Further improvements and bug fixes
• Block deletion is now more performant.
• Some inconsistencies in the UI have been removed.
• Error messaging is clearer in the deployment build
CLI command and the CLI got much more descriptive:
• Resolved timeout errors that occurred when using async task functions inside synchronous flows.
Discourse version with nicer formatting.
Happy Engineering! :marvin-duck:Anna Geller
08/02/2022, 11:17 PMprefect-dbt
! With this Prefect Collection, we’re now able to support common workflows for both:
1. dbt CLI for Open-Source dbt users
2. and dbt Cloud for commercial dbt users.
We worked with two solutions architects from dbt to develop requirements, QA strategy, and complete development for this collection to ensure this integration reflects best practices for using dbt! :dbt:
Kudos to @alex and @Andrew Huang for developing this awesome integration! 👏
If you would like to add a feature request for additions to this collection, feel free to add an item to the `prefect-dbt` GitHub project!justabill
08/03/2022, 8:15 PM2.0.0
release, today's 😛refect:`1.3.0` release is an important reminder that we will continue to support Prefect 1.x well into the future 💪. Today's release enables custom job name prefixes using the job template passed to the KubernetesAgent. It also includes several task library enhancements and fixes, including:
• :kubernetes: Target a specific container within a Kubernetes pod with the new ReadNamespacedPodLogs
parameter
• 🐚 Run commands in Microsoft Powershell with the new PowershellTask
• :azure: Interface with Azure Datafactory more confidently with several fixes to existing tasks
Special thanks to @Alvaro Durán Tovar and @Chinmay Bhat for their contributions to this release!
See the Prefect 1.x changelog for more information.Chris Reuter
08/05/2022, 4:30 PMJeff Hale
08/08/2022, 3:58 PM.prefectignore
• Resolved bugs and improved interface in the Orion client.
• Resolved a bug in Azure storage blocks that would cause uploads to get stuck.
• Resolved a bug where calling a flow in a separate thread would raise an exception.
• Resolved issues with loading flows from a deployment.
• Corrected some erroneous type annotations.
• Better handling of database errors during state transition validation.
• Better sanitization of labels for Kubernetes Jobs.
• Fixes --manifest-only
flag of prefect deployment build
command to ensure that using this flag, the manifest gets generated, but the upload to a storage location is skipped.
• Added support for multiple YAML deployment paths to the prefect deployment apply
command.
We want to highlight again that we released prefect-dbt
- a collection of Prefect integrations for working with dbt in your Prefect flows. This collection has been built as part of a partnership with dbt Labs to ensure that it follows best practices for working with dbt.
Thank you to everyone who contributed feedback, opened issues, wrote code, and improved documentation! 😛refect-duck:💙Anna Geller
08/10/2022, 6:43 PMmanifest.json
-- many users gave us feedback that having two manifests (YAML for infrastructure and JSON for flow path configuration) was confusing. We simplified that. Now, there is a single YAML manifest that gets generated as a build artifact anytime you run prefect deployment build
. You may version control that file or simply use it as a build artifact in your CI/CD (recipes for that are coming!).
2. The local file system block is handled behind the scenes to streamline the local development experience.
3. The infrastructure blocks have been optimized for modularity and composability. Instead of boilerplate code, you can now have a generic Process
, DockerContainer
or KubernetesJob
block, and add custom overrides in your build CLI only for changes (overrides) you need to apply on a per-deployment level. For example, if you want to change the default log level for a given deployment to DEBUG (or change any other environment variable) or swap the Docker image for a different one, you can do all that using the new --override
flag:
prefect deployment build flows/your_flow.py:flow_function_name \
-n deployment_name \ # name for this specific deployment
-t your_workqueue_name \ # tag corresponding to your work queue agent
-o deployments/deployment_manifest1.yaml \ # output: where to store your manifest
-ib process/dev \ # your generic parent Infrastructure Block to avoid boilerplate
--override env.PREFECT_LOGGING_LEVEL=DEBUG \
--override image=prefecthq/prefect:2-python3.9 \
-v your_github_sha # version: for robust deployment versioning from CI/CD
---
Start custom flow runs from the UI
One of the most commonly requested features was the ability to start parametrized flow runs directly from the UI. You can now do that!
From the deployment page, in addition to triggering an immediate flow run with default parameter arguments, you can also create a custom run allowing you to configure:
• the run's parameter values,
• start time,
• custom run name,
• and more,
All that is possible while using otherwise the same deployment configuration (infrastructure, storage, etc.). The deployment itself will be unchanged and continue to generate runs on its regular schedule. See the attached screenshot to see how this works.
---
Further improvements and bug fixes:
• Timeout errors messages on state changes are more intuitive
• Added debug level logs for task run rehydration
• Added basic CLI functionality to inspect Blocks; more to come. Here is how it looks like:
prefect block ls # list blocks in the workspace
prefect block type ls # list registered block types
prefect block register --help # shows how to register new block types
• Added support for filtering on state name to prefect flow-run ls
• Refined autogenerated database migration output
As always, to upgrade to the latest version:
pip install prefect -U
Cloud 2.0 is already up to date! :cloud2:
Happy engineering!Anna Geller
08/10/2022, 8:25 PMprefect-monday
is now released! :marvin-duck:
A fun tidbit about this integration is that the tasks within this Prefect Collection were created by a code generator using the service's GraphQL schema.
• Code
• Docs
Don’t forget to give this repository a ⭐!Anna Geller
08/11/2022, 6:12 PMAnna Geller
08/17/2022, 1:08 PMpip install prefect-databricks
Docs: https://prefecthq.github.io/prefect-databricks/
Code: https://github.com/PrefectHQ/prefect-databricks
If you like it, give us a star on this repository! ⭐Anna Geller
08/17/2022, 11:08 PMprefect deployment build
CLI provides a simple way of creating deployments, and it has been designed for reliable engineering workflows that integrate well with CI/CD systems such as GitHub Actions (tutorials and recipes follow soon!). However, some users requested an additional feature to also allow deployment creation purely using Python. See the Discourse announcement with an example of how you can use that deployment pattern. The documentation has been extended accordingly.
---
Simplified Agents & Work Queues
Agents and work queues give you control over where and how flow runs are executed. Now, creating an agent (and corresponding work queue) is even easier. Why?
• Work queues now operate strictly by name, not by matching tags. Tags remain an important concept for organizing your work based on projects and filtering your deployments and runs in the UI.
• Deployments, and the flow runs they generate, are explicitly linked to a single work queue - this means that deployments map 1:1 to work queues rather than via a many-to-many relationship.
• The work queue is automatically created whenever a deployment references it - this means that you no longer have to worry about creating a work queue manually; assign a new work queue name to your deployment when you need it, and just start a new agent polling from the same work queue name. Prefect will handle work-queue creation under the hood for you.
Here is what this looks like in practice:
prefect deployment build myflow.py:myfunc -n mydeployment -q prod -o my.yaml
prefect deployment apply my.yaml
prefect deployment run myfunc/mydeployment
prefect agent start -q prod
This “prod” work queue will pick up the run since it matches the queue -q prod
specified on this deployment.
Default work queue
If you forget a work queue name on your deployment, Prefect will automatically assign a “default” work-queue name. This way, the runs generated by such deployments can be picked up by an agent you can start using:
prefect agent start -q default
Benefits of the work queue changes
Most users will not have to interact directly with work queues at all – you only need to interact with deployments and agents. You can use work queues only once you need a more advanced configuration, such as using concurrency limits.
Work queue changes are backward compatible!
The changes are fully backward compatible. See the documentation to learn more.
---
New Collections
We announced those here already, but to list those again, here are recently released Prefect Collections:
• prefect-monday
• prefect-databricks
• prefect-fugue
As always, to upgrade to the latest version:
pip install prefect -U
Cloud 2.0 is already up to date! :cloud2:
Happy engineering! :marvin-duck:Chris Reuter
08/18/2022, 1:00 PMAnna Geller
08/18/2022, 3:38 PMAnna Geller
08/23/2022, 6:28 PMAnna Geller
08/24/2022, 1:51 PMAnna Geller
08/24/2022, 1:51 PM