Hey everyone! A few months ago, I posted a discussion on optimizing for high memory usage in the #best-practices-coordination-plane: https://prefect-community.slack.com/archives/C03D12VV4NN/p1671793942895089 We're still running into this issue. Idk if this apparent build up of memory usage is a bug with prefect or if it is something we can control

Getting this error sporadically on some of our flows:

Submission failed. botocore.exceptions.ParamValidationError: Parameter validation failed: Unknown parameter in input: "deregisteredAt", must be one of: family, taskRoleArn, executionRoleArn, networkMode, containerDefinitions, volumes, placementConstraints, requiresCompatibilities, cpu, memory, tags, pidMode, ipcMode, proxyConfiguration, inferenceAccelerators, ephemeralStorage, runtimePlatform

In the past I got this error when

prefect_api_key

or

prefect_api_url

was missing, but that is not the case anymore because the same flow/deployment runs fine other times.

hello! I'm brand new to prefect 2 and I'm running into some issues using the ECS block. I've created an ECS block but whenever I try to add any labels at all I get an error when trying to load said block. I've attached a screenshot of what I see in the UI and the error I get when the "Labels" field has a value in it. How can I create an ECS block that with these fields filled out that passes validation when trying to load that block?

Hi Max! Labels need to be entered in key-value format. For ECS tasks, the labels get added to the task as tags. So in your case, you might enter something like:

{
  "taskType": "prefect-agent"
}

You can choose just about any key you'd like; the important part is that you choose something that's meaningful to you.

Thank you so much Ryan! That worked wonderfully, thank you so much. I have another related question: I can't seem to find that block when attempting to deploy, i.e. with a command like

prefect deployment build flows/first_flow.py:hi -n dev -q dev -ib ecs/first-ecs-test -sb s3/first-flow -a

I get the attached error. am I using this block properly?

Hi all! New to prefect v2 and wrapping my head around the deployment for an ECSTask. Correct me if I’m wrong, but each flow run has to deploy a new task definition to run? How can I have more than one flow run at a time?

Hi, Is everyone using the AwsCredential block that well working in prefect 2.0? Why can't I keep checking the keys of objects in the bucket due to the SignatureDoesNotMatch problem? If i hard code(write the access key and secert access key) in the flow, it will work. If this is the case, isn't there something wrong with this block? I also asked in slack and also asked at https://discourse.prefect.io/, but no one answered.. Just this code!

import boto3
from prefect_aws import AwsCredentials

@flow
def check_conndition():
    ac_block = AwsCredentials.load('my_block')
    s3 = boto3.resource('s3', aws_access_key_id=ac_block.aws_access_key_id, aws_secret_access_key=ac_block.aws_secret_access_key)
    bucket = s3.Bucket('my_existing_bucket') 
    
    obj_list = []
    for obj_summary in bucket.objects.all(): #error occurring point
        obj_list.append(obj_summary.key)

if __name__=='__main__':
    check_condition()

Hi all! Does anyone know what you need to do to actually make use of the ECS Task’s

stream_output

flag? Ive got

_stream_output_=True

and

_configure_cloudwatch_logs_=True

. The cloudwatch logs look fine (see the screenshot), but the stream_output flag doesnt seem to do anything. Im running the simple

command=["echo", "hello world"],

example right from the docs too, nothing fancy going on. I can confirm it has access to all three logs:CreateLogStream, logs:CreateLogGroup, and logs:PutLogEvents permissions, but the logs never actually appear in the prefect flows logs, which was what I was expecting. Does anyone know what I’m missing?

We are in the process of setting up a Prefect agent service in ECS, and we are trying to understand the ports that are currently being utilized by Prefect. Can you please help us understand or point us to where this is documented?

Hi all! I’m trying to get my tasks deployed on ECS and running into the following error:

prefect_aws.ecs.TaskFailedToStart: ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve secret from asm: service call has been retried 5 time(s): failed to fetch secret arn:aws:secretsmanager:XXXXXXXX from secrets manager: RequestCanceled: request context canceled caused by: context deadline exceeded

File "/usr/local/lib/python3.10/site-packages/prefect_aws/ecs.py", line 1058, in _wait_for_task_start

I’ve gone through and made sure that the ecs service has: 1. Public IP assigned 2. Public subnets 3. Egress security group that allows all traffic Anything that I missed?

the secret I’m trying to pull is from secrets manager

Did some digging. Does prefect override my selected VPC subnets when running the task? https://github.com/PrefectHQ/prefect-aws/blob/main/prefect_aws/ecs.py#L924

Hi, everyone! I'm trying to solve this issue and dropped a message in prefect-community, but got no answers 😞 I'm not sure if this is the right channel to re-ask, but has anyone dealt with this problem? Thank you!

Hi! A very QQ - Does anyone know at where local PATH inside the

prefect-agent

container, the agent downloads the content of an S3 Block with flows for a deployment?

Hi all! I’m using S3 storage - how can I ensure that the storage path in S3 is cleared before doing another deployment?

Hello, i have found the prefect Terraform recipe for setting up the infrastructure on AWS. Is there also possible to get a recipe for CDK? Does someone already created this and can share? 🙂

Hello! I have a perhaps-basic question but I can't seem to find what I need in the docs so I'm asking it here: I want to run my flows and agents from AWS ECR/ECS - (I can't find a good step-by-step guide on how to do that, so if anyone has any good guide for that let me know). I have started with a simple hello world and want to run it on ECS. I have docker-ized this flow and pushed it up to ECR and pointed to that registry in a custom ECS block and have pointed to that ECS block when deploying my flow with

prefect deployment build flows/ecs-test-flow.py:main -n dev -q dev -ib ecs-task/first-ecs-test -sb s3/first-flow -a

I have also tried just using the generic

-i ecs-task

when deploying my flows, to no avail. In either case I get 'Submission failed. KeyError: "No class found for dispatch key 'ecs-task' in registry for type 'Block'.' when trying to create a quick run of the flow. Am I going about this wrong? How can I reliably deploy my flows to ECS?

Is there a way to globally configure result persistence to S3 for all flows & tasks in a workspace? I know how to persist results to S3 on an individual flow, but I was hoping to find a way to avoid duplicating this configuration across our deployed flows.

Hello, i have used the terraform receipe to create all the ressources in AWS. Also my deployment runs in Prefect Cloud. If I start a new Run, the aws ecs instance get the flow transmitted, flow starts running and fails than because of an Boto3 exception. if I have added the the profilename to "default", i get an error that the profile name is not available. If i remove the profilename I get an error "botocore.errorfactory.ClientException: An error occurred (ClientException) when calling the DescribeTaskDefinition operation: Unable to describe task definition." Can someone help me here out? Thanks CHristian

Hello all! We have an interesting issue with our agents in ECS are not releasing memory. Anyone have any similar experiences or workarounds? The logs do not indicate any shutdown (graceful or otherwise), but this is leaving many flow runs stuck in PENDING.

hey all....do we know of a way to add secrets to the container definition of the ECSTask block? I tried this, but it just creates a new container to the task.

task_definition={
                "containerDefinitions": [
                    {
                        "secrets": [
                            {
                                "name": i.envar_name,
                                "valueFrom": f"arn:aws:secretsmanager:{AWS_REGION}:{account_id}:secret:{i.secret_name}",
                            }
                            for i in self.secrets
                        ],
                        "name": slugified_flow_name,
                        "image": image_name
                    }
                ],
                "ephemeralStorage": {"sizeInGiB": self.ephemeral_storage_gb},
            }

Hi, I'm wondering what the status of the cli-prefect2-ecs-agent recipe is. I tried to follow it, but it seems the IAM roles defined in the task-description are invalid. Is there perhaps a blog post to accompany this or something?

How can I give flows that run on my AWS ECS Fargate a cluster EFS/EBS volume to temporarily write files to? I have tried using the

task_customizations

field of

ECSTask()

, but it seems like the fields I want to modify (`volumes`/`mountPoints`) are unsupported (see thread).

trying to submit a deployment with ECS task overrides as json patches.....here is the command

prefect deployment build example_flow.py:flow_1 -n base -sb s3/common-utils-dev-test/flow-group-1/flow-1 -ib ecs-task/common-utils-flow-1-dev-test --override 'task_customizations=[{"op": "replace", "path": "/overrides/cpu", "value": "512"}, {"op": "replace", "path": "/overrides/memory", "value": "1024"}]' -q prefect-v2-queue-dev-test -v dev --params '{"param_name": "param_value"}' -t common-utils -t flow-group-1 -a --cron '0 0 * * *'

It results in this infra override in deployments

{
  "task_customizations": "[{\"op\": \"replace\", \"path\": \"/overrides/cpu\", \"value\": \"512\"}, {\"op\": \"replace\", \"path\": \"/overrides/memory\", \"value\": \"1024\"}]"
}

however when I submit the flow run I get this error

Submission failed. pydantic.error_wrappers.ValidationError: 1 validation error for ECSTask task_customizations instance of JsonPatch expected (type=type_error.arbitrary_type; expected_arbitrary_type=JsonPatch)

anyone know why this would not work.

Cannot get this test to pass Flake8....any suggestions. noqua is not working.

@pytest.mark.usefixtures("ecs_mocks")
async def test_task_customizations_as_string(aws_credentials):

    task = ECSTask(
        aws_credentials=aws_credentials,
        memory=512,
        cpu=256,
        task_customizations='[{"op": "replace", "path": "/overrides/cpu", "value": "512"}, {"op": "replace", "path": "/overrides/memory", "value": "1024"}]',
    )  # type: ignore

    original_run_task = task._run_task
    mock_run_task = MagicMock(side_effect=original_run_task)
    task._run_task = mock_run_task

    await run_then_stop_task(task)

    overrides = mock_run_task.call_args[0][1].get("overrides")

    assert overrides["memory"] == "1024"
    assert overrides["cpu"] == "512"

I am trying to install prefect on ec2 (Amazon Linux) but running into the asyncpg related error. Does anyone have any solution for that

Hi all, I'm hoping someone can assist in solving this issue I'm experiencing when running a Prefect Agent on ECS. It seems that the ECS Task works fine when it's started by the ECS Service and will pick up any work that has been scheduled or triggered. However, after a couple of hours of the ECS Task waiting for work, it throws an error (see attached screenshots of Cloudwatch Logs). After this error is thrown, the ECS Task container stays active but does not pick up any work. This causes schedule flows to be late until I manually stop the container and let the ECS Service provision a new ECS Task container. This is not ideal as we have some flows scheduled for graveyard hours. Two questions: 1. Any idea what is causing this error? 2. Is there a configuration that I am missing that will stop the container when an error occurs so that the ECS Service can provision a fresh container?

has anybody else ever encountered this error? (

ListObjects: permission denied

) https://github.com/PrefectHQ/prefect/issues/9188 it's quite a painful one since it appears to happen very randomly and i have no idea how to get around it

happens when pulling flows down from s3