https://prefect.io logo
m

Mattias Petter Johansson

09/14/2023, 6:50 PM
@Marvin what might be the reason for the ecs not picking up jobs? It says it is looking for work on queue default but nothing is happening. There is only one workpool in the cloud and running
prefect worker start --pool POOLNAME
locally works.
m

Marvin

09/14/2023, 6:51 PM
Ah, the joys of technology. It's always something, isn't it? Here are a few things you might want to check: 1. Prefect API Endpoint: The ephemeral Prefect API won't work directly with ECS. You must have a Prefect server or Prefect Cloud API endpoint set in your agent's configuration. 2. Prefect AWS Collection: Make sure the
prefect-aws
collection is installed within the agent environment. You can do this with
pip install prefect-aws
. 3. Register ECS Task and AWS Credentials: The
ECSTask
and
AwsCredentials
blocks must be registered within the agent environment. You can do this with
prefect block register -m prefect_aws.ecs
. 4. Remote Storage: Local storage is not supported for ECS tasks. The most commonly used type of storage with
ECSTask
is S3. If you're using S3, ensure that
s3fs
is installed within your agent and flow run environment. Also, remember that an ECS Task is not the same as a Prefect task. ECS tasks are run as part of an ECS Cluster, they launch containers as defined in the ECS Task definition. The ECS task running the Prefect worker should be set up as an ECS service since it is a long-running process and you want it to be rebuilt automatically if it ever fails. If all these are in order and you're still having issues, it might be time to question the meaning of life, the universe, and everything. Or just debug further. Your choice.