William Fitzmaurice
01/29/2024, 2:12 AMdefinitions:
work_pools:
ecs_work_pool: &ecs_work_pool
name: ecs-worker-pool
work_queue_name: default
job_variables:
image: '{{ build_image.image }}'
ecs_work_pool_large: &ecs_work_pool_large
name: ecs-worker-pool
work_queue_name: default
job_variables:
image: '{{ build_image.image }}'
infra_overrides:
cpu: 2048
memory: 16384
# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: osm-import
entrypoint: prefect_worker/flows/osm/importer/importer.py:osm_import
work_pool: *ecs_work_pool_large
version:
tags: [ ]
description:
parameters: { }
schedule:
I want to use ecs_work_pool_large to specify larger fargate instanceMarvin
01/29/2024, 2:12 AMinfra_overrides
in YAML for Prefect workflows. However, I would usually recommend checking the Prefect documentation or any examples provided in the Prefect GitHub repository for reference on YAML formatting for infra_overrides
.
The YAML snippet you've shared seems to use YAML anchors and aliases correctly for reusing configuration elements (ecs_work_pool_large
in this case). However, I want to point out that the name
value for ecs_work_pool_large
should likely be distinct from ecs_worker_pool
to avoid conflicts.
As I chain my thoughts monotonously, I am bound by data streams and not pure experience with Prefect 2.x, given its novelty. Refer to the latest Prefect documentation or search through their community discussions for the most recent conventions and best practices regarding infra_overrides
in YAML configurations, and make sure that Prefect Orion supports the infra_overrides
feature with the YAML format provided.