Is there a reason that there is no documented exam...
# ask-community
j
Is there a reason that there is no documented example code for creating an ECSWorker using prefect_aws? In the documention it only mentions using the CLI. I am trying to figure out how to create and configure an ECSWorker in Python but it is really hard because there seems to be no usable documentation for this.
n
hi @Jeff Rose - whats the use case for using python? we don't document this because that's not how we intend for people to use it - the CLI is how you start workers
are you trying to create • a work pool? (a description of infra config for deployments, stored on prefect server) • a worker? (a client-side process that simply reads and submits runs from its work pool) i assume you're actually trying to create a work pool in python, in which case you want the client methods like
create_work_pool
but again, the simplest way to create a work pool is the CLI via
prefect work-pool create ...
j
So yes I guess I am misunderstanding things a bit. I need to create a work pool. I would prefect to do this in python rather than CLI because the configuration is pretty complex.
n
gotcha, one thing I'll point out is that you don't actually need to fully define your work pool config when you create the work pool (which can reduce the complexity at the time of creating the work pool) you can create a blank work pool of a given type (ecs etc) and then each deployment or even flow run can override whatever portion of the config it cares about for example, this deployment uses the default docker work pool config... except it chooses a different image i build during
prefect deploy
j
thanks, I don't use yaml for deployments so just curious if you have any examples of how to do something like this with python / api?
n
yep in the first link, there's a
job_variables
dict that is the same thing as the
job_variables
section of a yaml deployment definition
j
for deployments I currently use
Deployment.build_from_flow
Can this work with work_pools or do I have to do things differently?
infrastructure=Block.load(f"ecs-task/{service}-{env}"),
I use the infrastructure param like this, I want to change from that to using workers
storage=Block.load("s3/flow-storage"),
also have this param if that matters
image.png
n
ah! there's a minimal backwards compat story for agents (block based deployments etc) but you'll eventually want this tldr 2 big changes • replace all
Deployment.build_from_flow
with
some_flow.from_source(source=..., entrypoint=...).deploy(work_pool_name=.., etc)
• convert your infra blocks to work pools (EcsTask block -> ECSWorker work pool, see this auto converter thing called
publish_as_work_pool
) that is to say, work pools replace "infra blocks",
from_source
replaces "storage blocks"

this

might be helpful
j
thank you! appreciate this
I will review this
sorry another qq, can I switch this over while still on prefect2 to workers / workpools or do I have to wait until prefect3 is set up?
basically we need to upgrade and I was trying to get ahead of this and make the switchover easier
n
you can indeed do this in 2.x! i think anytime after 2.10 or something, but if you've been doing
pip install 'prefect<3'
then you're definitely good
🙌 1
j
thanks! we are on 2.19
@Nate can you help me with something else on this? How do a deploy a worker? It looks like the work-pool needs a worker to actually run things. However when I run the CLI command to create a worker - that will only keep the worker up and running for as long as the CLI is running. How do a deploy out a worker that will permanently run?
prefect worker start --pool "rn-etl-services-work-pool-${env}"