justabill
justabill
justabill
justabill
justabill
justabill
justabill
Jeff Hale
09/20/2023, 3:41 PMChris Reuter
09/21/2023, 1:00 PMSarah Krasnik
09/21/2023, 4:58 PMJeff Hale
09/21/2023, 8:53 PMprefect deploy -n flow-a/*
๐ ๏ธ Various fixes, including Prefect variable resolution in the deployments section of prefect.yaml
๐ Documentation updates and ๐ค new contributions from the community
See the release notes for more information. launchWill Raphaelson
09/21/2023, 9:12 PMJeff Hale
09/28/2023, 10:36 PMpull_steps
on Deployments page in the UI
๐ Documentation updates and ๐ค new contributions from the community
See the release notes for all the details. prefect rocketChris Reuter
10/03/2023, 5:28 PMJeff Hale
10/05/2023, 10:00 PMflow.from_source
method, you can specify a git repository and a path to a flow file in that repository. This method will return a flow object that can be run or served with flow.serve()
.
๐งช Experimental Pydantic 2 compatibility! Weโre working eagerly toward having Prefect installable with either pydantic<2
or pydantic>2
. As a first step toward compatibility, weโve ensured that Prefectโs use of Pydantic is isolated from your use of Pydantic in as many ways as possible.
As always this release includes lots of bug fixes ๐, documentation updates๐, and community contributions๐ค!
See the release notes for more details. prefect buildSarah Krasnik
10/10/2023, 8:08 PMSerina
10/12/2023, 9:23 PMPREFECT_EXPERIMENTAL_ENABLE_ENHANCED_CANCELLATION
feature. When enabled, you can cancel flow runs where cancellation can fail, such as when your worker is offline.
๐ฆ The new PREFECT_DEFAULT_RESULT_STORAGE_BLOCK
setting allows users to set a default storage block on a work pool or via job variables for a deployment. Itโs now easier to swap result storage locations by updating the environment in the UI or your prefect.yaml
file, eliminating the need to alter your flow source code.
See our release notes for more information. prefect spinChris Reuter
10/17/2023, 3:22 PMprefect-dbt-flow
that displays dbt model execution in the Prefect timeline view.
if you're in San Diego at dbt Coalesce this week make sure to stop by Booth 222 and grab one of these cool fuchsia ducks! We'll see you there dbt prefect rocketJeff Hale
10/19/2023, 9:10 PMflow.deploy
allows you to easily create a Docker image with your flow code baked in and use that image with the work pool of your choice.
Updating a Python serve script to a deploy script is as simple as changing serve
to deploy
, providing a work pool to deploy to, and providing a name for the built Docker image. See the ๐งต for an example.
As always, this release includes ๐ bug fixes, ๐๏ธ quality-of-life enhancements, ๐ documentation updates, and ๐ค new contributions from the community!
See the release notes for more information. prefect buildJeff Hale
10/25/2023, 7:44 PMprefect-client 2.14.0
. prefect rocket
Things to know about Prefect Client:
1. Installed size (with dependencies) is less than half the size of Prefect.
2. Itโs a stripped down version of the Prefect package without a server, UI, or CLI. Many dependencies are removed.
3. You can have both prefect
and prefect-client
in the same environment (for example if you thought prefect-client
would meet your needs but found you needed to install prefect
for additional functionality).
4. You import objects from prefect-client
exactly like you do from prefect
. For example: from prefect import flow
.
5. The Prefect Client is built out of the prefecthq/prefect GitHub repo and we plan to keep the versions of Prefect and Prefect Client in sync (the latest versions of both are currently 2.14.0).
6. We do not currently publish a Docker image that uses the Prefect Client.
prefect-client
is available today on PyPI, and can be installed with pip install prefect-client ๐
Please share any feedback about the Prefect Client in Slack. If you find bugs or desire additional features, feel free to open an issue in the Prefect GitHub repo. We plan to continue development it to make it as functional and light-weight as possible.
This release also includes bug fixes ๐ and documentation ๐ updates.
See the release notes for all the details. ๐Will Raphaelson
11/02/2023, 3:44 PMWill Raphaelson
11/02/2023, 10:48 PMflow.from_source
, and a host of bugfixes and docs improvements.
Deployment Status Monitoring ๐ฆ
โข Track deployment status in the Prefect UI
โข Emit Prefect Cloud events for deployment status changes, with automation triggers available
Expanded Storage for flow.from_source
๐๏ธ
โข Load flows from remote storage locations like S3 and Azure Blob Storage using fsspec
protocols
Enhancements and Fixes ๐ง
โข Skip Docker builds with flow.deploy
โข Fixes for cancellation cleanup, Pydantic V2 models in flow signatures
Documentation and Guides ๐
โข Adds documentation for Systemd daemonization, global concurrency clarification, and Prefect Cloud teams
prefect spin See the release notes for more details!Jeff Hale
11/09/2023, 10:51 PMDeploymentImage
for .deploy.
โข Allow image
passed into .deploy
to be optional if loading flow from storage.
โข Ensure client avoids image builds when deploying to managed work pools.
As always, bugs were squashed and documentation was improved. ๐๐
Check out all the details in the release notes. prefect rocketJeff Hale
11/13/2023, 5:52 PMJeff Hale
11/16/2023, 11:57 PMflow.from_source
for synchronizing credentials and configurations. See the example in the image below.
๐๏ธ Weโve added two new settings enhancements:
โข A client-side setting for specifying a default work pool.
โข Host and port settings for the worker webserver are now customizable.
๐ Weโve updated our work pool-based deployment guide to include information on creating deployments with .deploy
.
The release contains a number of other
๐ bug fixes
๐ documentation updates
๐ค new contributions from the community
Check out all the details in the release notes! prefect spinjustabill
justabill
justabill
prefect work-pool create
CLI command to automatically configure your GCP project and set up your Prefect workspace to use a new Cloud Run push pool immediately.
You can create a new Cloud Run push work pool and configure your project with the following command:
prefect work-pool create --type cloud-run:push --provision-infra my-pool
See the following pull request for implementation details: โ #11204justabill
justabill
prefect work-pool create --type ecs:push --provision-infra my-pool
Using the --provision-infra
flag will automatically set up your default AWS account to be ready to execute flows via ECS tasks:
โญโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฎ
โ Provisioning infrastructure for your work pool my-work-pool will require: โ
โ โ
โ - Creating an IAM user for managing ECS tasks: prefect-ecs-user โ
โ - Creating and attaching an IAM policy for managing ECS tasks: prefect-ecs-policy โ
โ - Storing generated AWS credentials in a block โ
โ - Creating an ECS cluster for running Prefect flows: prefect-ecs-cluster โ
โ - Creating a VPC with CIDR 172.31.0.0/16 for running ECS tasks: prefect-ecs-vpc โ
โฐโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฏ
Proceed with infrastructure provisioning? [y/n]: y
Provisioning IAM user
Creating IAM policy
Generating AWS credentials
Creating AWS credentials block
Provisioning ECS cluster
Provisioning VPC
Creating internet gateway
Setting up subnets
Setting up security group
Provisioning Infrastructure โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 100% 0:00:00
Infrastructure successfully provisioned!
Created work pool 'my-pool'!
If you haven't tried an ECS push pool yet, now is a great time!
Don't fret, Azure users; we will add support for Azure Container Instances push work pools soon! soon horse
See the release notes for information on the other enhancements and fixes in today's release.
Happy engineering! marvin duck