Hey everyone! I'm so close to deploying my first flow, with

ECSTask

infrastructure. I'm programmatically creating a

Deployment

for a simple flow and it seems to have registered on our Prefect Cloud. However, the deployment run fails without log messages in Prefect Cloud. The agent logs

botocore.exceptions.ParamValidationError

, which makes me think this might be out of my reach. I'm currently in what seems to be the exact same situation as @Anthony Desmier (link to thread) is in. Not really sure where to go next. Any help is appreciated a lot, thank you for your time!

Hi Kevin, looks like the latest available version via install for pip is 0.1.4 - I’ll check with the integrations team on this

Just updated the deployed images and now it is provisioning the ECS tasks! Thank you!

Haven't had a chance yet to update to the latest release but this is great news!

Unfortunately, I think there's still something weird. From the logs of the prefect agent: • Agent retrieves task definition (from ARN) • Registers a task definition • Creates task runs with new definition Checking the ECS Task Definition reveals a new revision which has an additional container

prefect

(

prefecthq/prefect:2.6.1-python3.10

) Back to agent, logs: • Task run moves from

PROVISIONING

,

PENDING

, and finally

RUNNING

• Runs

python -m prefect.engine

in the

prefect

container • Status moves from

DEPROVISIONING

to

STOPPED

•

prefect

container exited with non-zero exit code

143

I'm not sure if this is the intended behavior

@Kelvin DeCosta @Chris Gunderson @Anna Geller same issue here, attached logs of this:

We have version 0.1.6 for prefect-aws in our docker image.

I worked around this by specifying a task definition (by ARN) that had one container called

prefect

which was built from an image with

prefect

and the modules required for our flows. Also, I explicitly specified that the

image

field for the

ECSTask

must be set to

None

, otherwise it defaults to using a barebones

prefect

image.

@Kelvin DeCosta I feel I'm getting the same errors you are with regards to the AccessDenied error on the S3 storage bucket. My flow container image is defined on the ECS task and I'm passing None for the image parameter on the ECSTask block. When I create a deployment, the files are successfully uploaded to the S3 bucket so the storage block itself does have the right permissions. The flow is successfully provisioned via the agent but the error on the flow container is

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

It's hard to tell if this is using the right storage block I defined in the deployment or whether I need to pass the permissions differently? Wondering if you could provide insight into how you got round it. Thanks!

Hey @Anthony Desmier I remember getting the S3 related Access Denied error only when creating a deployment and storing it on S3. I've since changed our setup to use our private

GitHub

repo instead. It's a win - win because we store our code on GitHub anyway, and don't have to deal with S3 That being said, I still had to update the permissions for the IAM user I created specifically for Prefect ( for the

AwsCredentials

block). I realized that the flow container was not able to access the S3 objects because I hadn't specified

S3FullAccess

as one of its policies.

Thanks Kelvin. That's interesting about using GitHub for storing. Might be worth us exploring that option. I'll double check my IAM polices and see if anything is untoward. Will post any solutions I find back here 👍

@Kelvin DeCosta did you use a custom image then? Also, after all this, you no longer get a 143 exit code?

Hey @chicago-joe Yes I did use a custom Image. Initially, I had a Task definition that used one container (the default, named

container

) which had all our flows and dependencies installed (including

prefect

) When manually running the deployments from the UI, I kept getting

143

. I checked the ECS AWS console and noticed that there were many revisions for the task definition I had created. I thought this was normal since I was using IaC (creating resources via Pulumi / Terraform). I then checked the Prefect Agent logs on Cloudwatch. They showed that there were new task definitions being registered by the agent. These new revisions had another container called

prefect

which I'm assuming is how the task execution engine runs. After poking through the

prefect

code base on GtiHub I realized that: • I could name my default container

prefect

(the one based on the image with all our flows + dependencies) and supply any necessary env variables • I should explicitly set

image

to

None

when creating the

ECSTask

infrastructure block

@Kelvin DeCosta omfg you're a legend this is what we had (task definition default with name

flow

)

# Task definition
containerDefinitions:
- image: spiderrockadvisors/orion:0.0.1
  name: flow
- image: prefecthq/prefect:2.6.4-python3.9
  logConfiguration:
    logDriver: awslogs
    options:
      awslogs-create-group: 'true'
      awslogs-group: prefect
      awslogs-region: us-east-2
      awslogs-stream-prefix: prefect
  name: prefect
cpu: '1024'
executionRoleArn: arn:aws:iam::393312900658:role/ecsworker_ecs_execution_role
family: prefect
memory: '2048'
networkMode: awsvpc
requiresCompatibilities:
- FARGATE

when changing the name to `prefect`:

# Task definition
containerDefinitions:
- image: spiderrockadvisors/orion:0.0.1
  logConfiguration:
    logDriver: awslogs
    options:
      awslogs-create-group: 'true'
      awslogs-group: prefect
      awslogs-region: us-east-2
      awslogs-stream-prefix: prefect
  name: prefect
  repositoryCredentials:
cpu: '1024'
executionRoleArn: arn:aws:iam::393312900658:role/ecsworker_ecs_execution_role
family: prefect
memory: '2048'
networkMode: awsvpc
requiresCompatibilities:
- FARGATE

@chicago-joe, not too sure... is it involving the path

/opt/prefect/flows

?