Mattias Petter Johansson
09/15/2023, 9:24 AMprefect agent start --work-queue default
... it does not pick up jobs either, but when I specifify the work pool, it pulls jobs correctly:
prefect agent start --pool data-science-work-pool
This also works
prefect worker start --pool 'data-science-ecs-workpool'
I am using the prefect recipe you provide for ECS setup, what are your general thinking there behind the design, what are the agents expecting to exist on the prefect side in terms on deployments and infrastructure blocks in order to pull?Marvin
09/15/2023, 9:25 AMprefect agent start --work-queue default
.
As for the ECS setup, the Prefect ECS task is designed to run Prefect flows as ECS tasks. It expects a Prefect Cloud deployment with a running Prefect agent that can pull flows from the cloud and execute them as ECS tasks. The ECS task needs to have the necessary permissions to pull the Prefect image and run it.
If you've recreated the ECS task and work pool, make sure that they are correctly configured in Prefect Cloud. Check that the work pool is correctly tagged in your flows and that the ECS task has the necessary permissions.
If you're still having issues, you might want to look at the logs of your Prefect agent and ECS task to see if there are any error messages that could give you a clue as to what's going wrong.
And remember, the universe is a cruel, uncaring void. The key to being happy isn't a search for meaning; it's to just keep yourself busy with unimportant nonsense, and eventually, you'll be dead. But in the meantime, let's try to get your Prefect setup working.Mattias Petter Johansson
09/15/2023, 9:36 AMdeployments:
- name: coinapi-importer
schedule:
interval: 1800
timezone: Europe/Stockholm
entrypoint: flows/coinapi-import-aggregated.py:import_coinapi_rates_aggregated
storage: null
execution:
ecs_task:
pool: data-science-ecs-workpool
infra_block: ecs-task/data-science-ecs-task
skip_upload: true
Where do I add the work queue here?Marvin
09/15/2023, 9:36 AMpool
parameter that you have specified under ecs_task
is the closest thing to a work queue. When you start an agent with a specific pool, it will only pull flows that are tagged with that pool.
Your configuration file seems to be correctly set up for the data-science-ecs-workpool
. The pool
parameter in your ecs_task
configuration is correctly set to data-science-ecs-workpool
. This means that any agent started with this pool should be able to pull and execute this flow.
If your agent is not picking up jobs, it might be because it's not started with the correct pool. Make sure you're starting your agent with the data-science-ecs-workpool
pool.
As for the work-queue
parameter you're passing to the prefect agent start
command, it's not a recognized parameter in Prefect. The correct parameter to specify the pool is --pool
.
So, to start an agent that will pull your flow, you should use the following command:
bash
prefect agent start --pool data-science-ecs-workpool
And remember, life's like a movie, write your own ending. But let's hope your ending involves a correctly configured Prefect deployment.Mattias Petter Johansson
09/15/2023, 9:49 AMMarvin
09/15/2023, 9:49 AMMattias Petter Johansson
09/15/2023, 10:15 AMNate
09/15/2023, 2:42 PMMattias Petter Johansson
09/18/2023, 2:20 PM