Kelvin DeCosta
10/17/2022, 5:03 PMECSTask
infrastructure.
I'm programmatically creating a Deployment
for a simple flow and it seems to have registered on our Prefect Cloud.
However, the deployment run fails without log messages in Prefect Cloud.
The agent logs botocore.exceptions.ParamValidationError
, which makes me think this might be out of my reach.
I'm currently in what seems to be the exact same situation as @Anthony Desmier (link to thread) is in.
Not really sure where to go next.
Any help is appreciated a lot, thank you for your time!AWSCredentials
• ECSTask
infra, created by specifying the cluster and task definition ARN
• S3
storage
Deployments are created using Deployment.build_from_flow
and passing the ECSTask
as infra.AccessDenied
error when specifying the S3
bucket storage.
I specified the bucket_path
as just the bucket_name
alone, which I'm starting to think is wrong.prefect-aws==0.1.5
seems to have addressed this issue.
However, it doesn't seem to be available for install via pip
.
Any updates on when it will be available?Christopher Boyd
10/17/2022, 5:51 PMKelvin DeCosta
10/18/2022, 3:02 PMAnthony Desmier
10/18/2022, 3:06 PMKelvin DeCosta
10/18/2022, 3:19 PMprefect
(prefecthq/prefect:2.6.1-python3.10
)
Back to agent, logs:
• Task run moves from PROVISIONING
, PENDING
, and finally RUNNING
• Runs python -m prefect.engine
in the prefect
container
• Status moves from DEPROVISIONING
to STOPPED
• prefect
container exited with non-zero exit code 143
I'm not sure if this is the intended behaviorchicago-joe
10/26/2022, 9:03 PMChris Gunderson
10/26/2022, 9:15 PMKelvin DeCosta
10/27/2022, 9:00 AMprefect
which was built from an image with prefect
and the modules required for our flows.
Also, I explicitly specified that the image
field for the ECSTask
must be set to None
, otherwise it defaults to using a barebones prefect
image.Anthony Desmier
10/27/2022, 11:04 AMbotocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied
It's hard to tell if this is using the right storage block I defined in the deployment or whether I need to pass the permissions differently? Wondering if you could provide insight into how you got round it. Thanks!Kelvin DeCosta
10/27/2022, 12:21 PMGitHub
repo instead.
It's a win - win because we store our code on GitHub anyway, and don't have to deal with S3
That being said, I still had to update the permissions for the IAM user I created specifically for Prefect ( for the AwsCredentials
block).
I realized that the flow container was not able to access the S3 objects because I hadn't specified S3FullAccess
as one of its policies.Anthony Desmier
10/27/2022, 1:04 PMchicago-joe
10/27/2022, 1:07 PMKelvin DeCosta
10/27/2022, 1:38 PMcontainer
) which had all our flows and dependencies installed (including prefect
)
When manually running the deployments from the UI, I kept getting 143
.
I checked the ECS AWS console and noticed that there were many revisions for the task definition I had created.
I thought this was normal since I was using IaC (creating resources via Pulumi / Terraform).
I then checked the Prefect Agent logs on Cloudwatch.
They showed that there were new task definitions being registered by the agent.
These new revisions had another container called prefect
which I'm assuming is how the task execution engine runs.
After poking through the prefect
code base on GtiHub I realized that:
• I could name my default container prefect
(the one based on the image with all our flows + dependencies) and supply any necessary env variables
• I should explicitly set image
to None
when creating the ECSTask
infrastructure blockchicago-joe
10/27/2022, 5:50 PMflow
)
# Task definition
containerDefinitions:
- image: spiderrockadvisors/orion:0.0.1
name: flow
- image: prefecthq/prefect:2.6.4-python3.9
logConfiguration:
logDriver: awslogs
options:
awslogs-create-group: 'true'
awslogs-group: prefect
awslogs-region: us-east-2
awslogs-stream-prefix: prefect
name: prefect
cpu: '1024'
executionRoleArn: arn:aws:iam::393312900658:role/ecsworker_ecs_execution_role
family: prefect
memory: '2048'
networkMode: awsvpc
requiresCompatibilities:
- FARGATE
when changing the name to `prefect`:
# Task definition
containerDefinitions:
- image: spiderrockadvisors/orion:0.0.1
logConfiguration:
logDriver: awslogs
options:
awslogs-create-group: 'true'
awslogs-group: prefect
awslogs-region: us-east-2
awslogs-stream-prefix: prefect
name: prefect
repositoryCredentials:
cpu: '1024'
executionRoleArn: arn:aws:iam::393312900658:role/ecsworker_ecs_execution_role
family: prefect
memory: '2048'
networkMode: awsvpc
requiresCompatibilities:
- FARGATE
no such file or directory
error when I try to run one of these using the s3 storage block. any ideas?Kelvin DeCosta
10/28/2022, 8:32 AM/opt/prefect/flows
?