https://prefect.io logo
Title
k

Kelvin DeCosta

10/17/2022, 5:03 PM
Hey everyone! I'm so close to deploying my first flow, with
ECSTask
infrastructure. I'm programmatically creating a
Deployment
for a simple flow and it seems to have registered on our Prefect Cloud. However, the deployment run fails without log messages in Prefect Cloud. The agent logs
botocore.exceptions.ParamValidationError
, which makes me think this might be out of my reach. I'm currently in what seems to be the exact same situation as @Anthony Desmier (link to thread) is in. Not really sure where to go next. Any help is appreciated a lot, thank you for your time!
1
For more context: AWS Resources (via Pulumi): • Agent as an ECS Service (seems to be receiving deployment runs) • ECS Task Definition for our flows. • S3 Bucket Prefect Blocks: •
AWSCredentials
ECSTask
infra, created by specifying the cluster and task definition ARN •
S3
storage Deployments are created using
Deployment.build_from_flow
and passing the
ECSTask
as infra.
On a slightly unrelated note, I ran into an
AccessDenied
error when specifying the
S3
bucket storage. I specified the
bucket_path
as just the
bucket_name
alone, which I'm starting to think is wrong.
From what I understand,
prefect-aws==0.1.5
seems to have addressed this issue. However, it doesn't seem to be available for install via
pip
. Any updates on when it will be available?
c

Christopher Boyd

10/17/2022, 5:51 PM
Hi Kevin, looks like the latest available version via install for pip is 0.1.4 - I’ll check with the integrations team on this
Hi Kevin, this should be fixed!
k

Kelvin DeCosta

10/18/2022, 3:02 PM
Just updated the deployed images and now it is provisioning the ECS tasks! Thank you!
a

Anthony Desmier

10/18/2022, 3:06 PM
Haven't had a chance yet to update to the latest release but this is great news!
k

Kelvin DeCosta

10/18/2022, 3:19 PM
Unfortunately, I think there's still something weird. From the logs of the prefect agent: • Agent retrieves task definition (from ARN) • Registers a task definition • Creates task runs with new definition Checking the ECS Task Definition reveals a new revision which has an additional container
prefect
(
prefecthq/prefect:2.6.1-python3.10
) Back to agent, logs: • Task run moves from
PROVISIONING
,
PENDING
, and finally
RUNNING
• Runs
python -m prefect.engine
in the
prefect
container • Status moves from
DEPROVISIONING
to
STOPPED
prefect
container exited with non-zero exit code
143
I'm not sure if this is the intended behavior
c

chicago-joe

10/26/2022, 9:03 PM
@Kelvin DeCosta @Chris Gunderson @Anna Geller same issue here, attached logs of this:
c

Chris Gunderson

10/26/2022, 9:15 PM
We have version 0.1.6 for prefect-aws in our docker image.
k

Kelvin DeCosta

10/27/2022, 9:00 AM
I worked around this by specifying a task definition (by ARN) that had one container called
prefect
which was built from an image with
prefect
and the modules required for our flows. Also, I explicitly specified that the
image
field for the
ECSTask
must be set to
None
, otherwise it defaults to using a barebones
prefect
image.
:gratitude-thank-you: 1
👍 1
a

Anthony Desmier

10/27/2022, 11:04 AM
@Kelvin DeCosta I feel I'm getting the same errors you are with regards to the AccessDenied error on the S3 storage bucket. My flow container image is defined on the ECS task and I'm passing None for the image parameter on the ECSTask block. When I create a deployment, the files are successfully uploaded to the S3 bucket so the storage block itself does have the right permissions. The flow is successfully provisioned via the agent but the error on the flow container is
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
It's hard to tell if this is using the right storage block I defined in the deployment or whether I need to pass the permissions differently? Wondering if you could provide insight into how you got round it. Thanks!
k

Kelvin DeCosta

10/27/2022, 12:21 PM
Hey @Anthony Desmier I remember getting the S3 related Access Denied error only when creating a deployment and storing it on S3. I've since changed our setup to use our private
GitHub
repo instead. It's a win - win because we store our code on GitHub anyway, and don't have to deal with S3 That being said, I still had to update the permissions for the IAM user I created specifically for Prefect ( for the
AwsCredentials
block). I realized that the flow container was not able to access the S3 objects because I hadn't specified
S3FullAccess
as one of its policies.
a

Anthony Desmier

10/27/2022, 1:04 PM
Thanks Kelvin. That's interesting about using GitHub for storing. Might be worth us exploring that option. I'll double check my IAM polices and see if anything is untoward. Will post any solutions I find back here 👍
c

chicago-joe

10/27/2022, 1:07 PM
@Kelvin DeCosta did you use a custom image then? Also, after all this, you no longer get a 143 exit code?
k

Kelvin DeCosta

10/27/2022, 1:38 PM
Hey @chicago-joe Yes I did use a custom Image. Initially, I had a Task definition that used one container (the default, named
container
) which had all our flows and dependencies installed (including
prefect
) When manually running the deployments from the UI, I kept getting
143
. I checked the ECS AWS console and noticed that there were many revisions for the task definition I had created. I thought this was normal since I was using IaC (creating resources via Pulumi / Terraform). I then checked the Prefect Agent logs on Cloudwatch. They showed that there were new task definitions being registered by the agent. These new revisions had another container called
prefect
which I'm assuming is how the task execution engine runs. After poking through the
prefect
code base on GtiHub I realized that: • I could name my default container
prefect
(the one based on the image with all our flows + dependencies) and supply any necessary env variables • I should explicitly set
image
to
None
when creating the
ECSTask
infrastructure block
:gratitude-thank-you: 1
👍 1
c

chicago-joe

10/27/2022, 5:50 PM
@Kelvin DeCosta omfg you're a legend this is what we had (task definition default with name
flow
)
# Task definition
containerDefinitions:
- image: spiderrockadvisors/orion:0.0.1
  name: flow
- image: prefecthq/prefect:2.6.4-python3.9
  logConfiguration:
    logDriver: awslogs
    options:
      awslogs-create-group: 'true'
      awslogs-group: prefect
      awslogs-region: us-east-2
      awslogs-stream-prefix: prefect
  name: prefect
cpu: '1024'
executionRoleArn: arn:aws:iam::393312900658:role/ecsworker_ecs_execution_role
family: prefect
memory: '2048'
networkMode: awsvpc
requiresCompatibilities:
- FARGATE
when changing the name to `prefect`:
# Task definition
containerDefinitions:
- image: spiderrockadvisors/orion:0.0.1
  logConfiguration:
    logDriver: awslogs
    options:
      awslogs-create-group: 'true'
      awslogs-group: prefect
      awslogs-region: us-east-2
      awslogs-stream-prefix: prefect
  name: prefect
  repositoryCredentials:
cpu: '1024'
executionRoleArn: arn:aws:iam::393312900658:role/ecsworker_ecs_execution_role
family: prefect
memory: '2048'
networkMode: awsvpc
requiresCompatibilities:
- FARGATE
🎉 2
:gratitude-thank-you: 1
🙌 1
@Kelvin DeCosta I'm still getting a
no such file or directory
error when I try to run one of these using the s3 storage block. any ideas?
k

Kelvin DeCosta

10/28/2022, 8:32 AM
@chicago-joe, not too sure... is it involving the path
/opt/prefect/flows
?