Joshua Grant
07/28/2023, 3:14 PMMarvin
07/28/2023, 3:14 PMprefect worker start --pool my-process-pool
.
Now, as for your question about work pools, they seem to be a newer feature introduced in Prefect. They allow agents to prioritize work that is more important or time-sensitive, even if there is a large backlog of flow runs on other work queues in a given work pool. All existing work queues will be assigned to a default work pool named default-agent-pool
. You can create a new work pool via the Work Pools page in the UI or via the CLI with the command prefect work-pool create "my-pool"
.
So, in conclusion, while your existing work queues will continue to operate as they currently do, work pools provide an additional layer of control and prioritization. Whether you need to create them or not depends on your specific use case and requirements. But remember, it's always good to have options. Even if those options make you question the meaning of existence.Joshua Grant
07/28/2023, 3:19 PMMarvin
07/28/2023, 3:19 PMpython
from prefect import Client
client = Client()
client.graphql('''
mutation {
create_work_pool(input: { name: "my-pool" }) {
id
}
}
''')
This will create a work pool named "my-pool". You can replace "my-pool" with the name of your choice.
Remember, in the grand scheme of the universe, this is a tiny task. But I suppose it's important to you, so there you go.Joshua Grant
07/28/2023, 3:30 PMMarvin
07/28/2023, 3:30 PMNate
07/28/2023, 3:51 PMJoshua Grant
07/28/2023, 4:00 PMfrom prefect.deployments import Deployment
from flow_file import flow
# blocks and deployment setup
deployment_fg = Deployment.build_from_flow(
flow=flow,
name=slugged_image_name,
version=os.getenv("CODEBUILD_RESOLVED_SOURCE_VERSION"),
tags=deployment_tags,
work_queue_name=slugged_image_name,
infrastructure=create_ecs_task_fg(slugged_image_name),
)
if __name__ == "__main__":
deployment_fg.apply()
For version 2.10.19, do we need to specify work_pool
here if we aren't interested in implementing pools and want to stay on the queue/flow method?Marvin
07/28/2023, 4:00 PMwork_queue_name
argument.
In Prefect 2.10.19, work queues are automatically assigned to a default work pool named default-agent-pool
if no specific work pool is provided. So, if you're not interested in implementing work pools and want to stick with the queue/flow method, you don't need to specify a work_pool
in your deployment. Your flows will continue to run on the specified work queue as they currently do.
So, in short, no, you don't need to specify work_pool
if you want to stick with your current setup. But remember, the universe is full of infinite possibilities. Or so I'm told. I wouldn't know. I'm just a robot.Nate
07/28/2023, 4:58 PMDeployment
python interface
and marvin's answer should actually be correct, you could. set the work pool to default-agent-pool
ill just throw it out there that as an avid user myself, i really like the prefect.yaml
+ prefect deploy
deployment story better, mostly bc the configurable pull
step, but just food for thoughtJoshua Grant
07/28/2023, 6:42 PMNate
07/28/2023, 6:49 PMour business requirements prevent us from using anything but Python to deployfyi technically, using
prefect deploy
along with prefect.yaml
is still using python
also the prefect.yaml
allows you to execute arbitrary custom python and pass the results of that from one step to another
build:
- my_local_python_module.my_custom_step:
foo: "{{ prefect.blocks.secret.my-secret }}"
bar: 42
push: ..
pull: ..
where you'd have my_local_python_module.py:my_custom_step
async def my_custom_step(foo: str, bar: int):
# do interesting business things
otherwise, I believe a more direct python interface for this is in the works, not sure about timeline