Chris Reuter
11/05/2021, 3:11 PMjustabill
flow-run
and deployment
command-line interfaces
* Clarification of existing documentation and additional new documentation
* Fixes for database creation and startup issues
Learn more in the latest release notes.justabill
Chris White
Chris White
Chris Reuter
11/17/2021, 6:01 PMChris Reuter
11/18/2021, 7:39 PMjustabill
justabill
justabill
pip install -U "prefect>=2.0.0a"
, run some flows, and share your feedback in #prefect-community. We are continuing to design features for Radar, including the ability to monitor data, artifacts, and progress as the system processes, and we welcome your input.Chris Reuter
11/30/2021, 6:17 PMAndrew Black
12/07/2021, 5:58 PMChris Reuter
12/08/2021, 2:00 PMChris Reuter
12/10/2021, 9:00 PMAndrew Black
12/14/2021, 8:34 PMChris White
Chris Reuter
12/17/2021, 1:03 PMChris Reuter
12/22/2021, 7:57 PMChris Reuter
12/23/2021, 8:00 PMjustabill
Executor
to TaskRunner
🏃, this release introduces FlowRunner
🏃♂️, an analogous concept that specifies the infrastructure that a flow runs on. Just as a task runner can be specified for a flow, which encapsulates tasks, a flow runner can be specified for a deployment, which encapsulates a flow. This release includes two flow runners, which we expect to be the most commonly used:
• SubprocessFlowRunner - The subprocess flow runner is the default flow runner. It allows for specification of a runtime Python environment with virtualenv
and conda
support.
• DockerFlowRunner - Executes the flow run in a Docker container. The image, volumes, labels, and networks can be customized. From this release on, Docker images for use with this flow runner will be published with each release.
Future releases will introduce runners for executing flows on Kubernetes and major cloud platform's container compute services (e.g. AWS ECS, Google Cloud Run).
To learn more about flow runners and their relationship to deployments, check out their respective concepts docs themoreyouknow!Chris Reuter
01/18/2022, 7:49 PMjustabill
justabill
pip install prefect==1.0rc1
to try it out. The release is also available in a Docker image at prefecthq/prefect:1.0rc1-python3.8
.
Highlights of this release include:
• API keys are the way to authenticate with Prefect Cloud. Support for tokens has been removed. Check the API keys docs for details on moving to keys.
• Flow Environments
have been removed. Use RunConfig
instead to define where and how a flow run should be executed. We deprecated Environments
back in 0.14, but if you’re still using them, you won’t get a warning anymore. We have a guide on upgrading to RunConfig
if you need it.
• We’re dropping explicit support for Python 3.6.
There are some breaking changes in this release. You can read more about them in the changelog. Otherwise, check out the 1.0 documentation preview or join the discussion on Github. Soon we’ll have topics on migrating to Prefect 1.0 as well. We’re excited about this milestone and thankful to have your support through the process. 🚀Chris Reuter
01/26/2022, 4:21 PMAnna Geller
justabill
asyncio
(and anyio
) support, harnessing the power of concurrency without complex software patterns such as thread pools. The concurrent task runner enables tasks that are blocked on input/output to yield to other tasks, whether your code is synchronous or asynchronous.
When running a flow using a task runner that enables concurrent execution, or running many flows across multiple execution environments, you may want to limit the number of certain tasks that can run at the same time. For example, perhaps you want to ensure that, across all of your flows, there are no more than three open connections to your production database at once. You can do so by creating a “prod-db” tag and applying it to all of the tasks that open a connection to that database. Then, you can create a concurrency limit with prefect concurrency-limit create prod-db 3
. Now, Orion will ensure that no more than 3 task runs with the “prod-db” tag will run at the same time. Previously, this feature was only available in a paid tier of Prefect Cloud, our hosted commercial offering. We’re very happy to move it to the open source domain, furthering our goal of making Orion the most capable workflow orchestration tool ever 💪.
There is so much more in this release, including greater flow parameter flexibility, database migrations, CLI refinements, and much more. Read all about it in the release notes!
🚨 Breaking change: If you have previously installed any version of Orion, you'll need to delete existing database with rm ~/.prefect/orion.db
or stamp it with prefect orion database stamp
to successfully upgrade.Chris Reuter
02/10/2022, 4:29 PMAnthony Head
02/15/2022, 2:00 PMjustabill
prefect orion kubernetes-manifest
, that you can use to automatically generate a manifest that runs the Orion API, UI, and agent as kubernetes deployments.
With the Kubernetes flow runner, you can now run flows as Kubernetes Jobs. If you're running Orion in Kubernetes, you don't need to configure any networking. When the agent runs your deployment, it will create a job, which will start a pod, which creates a container, which runs your flow.
Check out the release notes or the tutorial to learn more about running Orion and flows on kubernetes .