Anna Geller
08/17/2022, 11:08 PMprefect deployment build
CLI provides a simple way of creating deployments, and it has been designed for reliable engineering workflows that integrate well with CI/CD systems such as GitHub Actions (tutorials and recipes follow soon!). However, some users requested an additional feature to also allow deployment creation purely using Python. See the Discourse announcement with an example of how you can use that deployment pattern. The documentation has been extended accordingly.
---
Simplified Agents & Work Queues
Agents and work queues give you control over where and how flow runs are executed. Now, creating an agent (and corresponding work queue) is even easier. Why?
• Work queues now operate strictly by name, not by matching tags. Tags remain an important concept for organizing your work based on projects and filtering your deployments and runs in the UI.
• Deployments, and the flow runs they generate, are explicitly linked to a single work queue - this means that deployments map 1:1 to work queues rather than via a many-to-many relationship.
• The work queue is automatically created whenever a deployment references it - this means that you no longer have to worry about creating a work queue manually; assign a new work queue name to your deployment when you need it, and just start a new agent polling from the same work queue name. Prefect will handle work-queue creation under the hood for you.
Here is what this looks like in practice:
prefect deployment build myflow.py:myfunc -n mydeployment -q prod -o my.yaml
prefect deployment apply my.yaml
prefect deployment run myfunc/mydeployment
prefect agent start -q prod
This “prod” work queue will pick up the run since it matches the queue -q prod
specified on this deployment.
Default work queue
If you forget a work queue name on your deployment, Prefect will automatically assign a “default” work-queue name. This way, the runs generated by such deployments can be picked up by an agent you can start using:
prefect agent start -q default
Benefits of the work queue changes
Most users will not have to interact directly with work queues at all – you only need to interact with deployments and agents. You can use work queues only once you need a more advanced configuration, such as using concurrency limits.
Work queue changes are backward compatible!
The changes are fully backward compatible. See the documentation to learn more.
---
New Collections
We announced those here already, but to list those again, here are recently released Prefect Collections:
• prefect-monday
• prefect-databricks
• prefect-fugue
As always, to upgrade to the latest version:
pip install prefect -U
Cloud 2.0 is already up to date! :cloud2:
Happy engineering! :marvin-duck: