Hi! :blob-attention-gif: First of all, thank you for the excellent template on integrating Prefect with AWS. As I understand it, it is now possible to run all ECS tasks only with a specific CPU and Memory, which we prescribe in the ECS Block. But is it possible to configure the CPU and Memory for each task separately? Because for some tasks I need min capacities, for others I need maximum ones.

This might be a dumb question, Is it possible to deploy my image as an ECS task without using infrastructure blocks?

Ok, hopefully my last question for ironing out our setup, as of this morning I'm seeing the error in the thread when deploying a flow

hey guys, I’m trying to

prefect deployment build

with

-sb

on S3 and

-ib

of a DockerContainer, then I got

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

I guess it has to do with the IAM role, but I don’ know how to switch/use a IAM role during

deployment build

, any ideas?

Is anyone else experiencing this issue where it does not delete task-def, even when

auto_deregister_task_definition

set to True? Is it normal for a deployment to create a new task-definition every time it runs an ECS task?

Hi, I’m trying to sending alarm to slack when my flow failed. Normal case it works well, but when flow about lack of aws resource (i.e., gpu) fails i didnt’t get the alarm. Now I’m using ECS agnet of prefect 0.15.3. Anyone could help?

Hi, This is kind of difficult to explain. I have a flow that performs a job over an EC2 Instance, and the problem is that sometimes it fails and the most of times, it succeed. The job that runs, do the following operations: 1- Download a jar from somewhere. 2- Download some script from github. 3- Execute the script with some sort of arguments. And what it does, is download data from AWS S3, process the files downloaded, and then upload the results. The errors that I usually got are like •

Main class from jar could not be found

•

The output directory does not exists, so it could not be synchronize to AWS S3

Do you have any idea of why these types of issues happens time to time? BTW: I am using

boto3

SSM

agent to handle EC2 Instances creation, execution and termination

Hey prefect-aws, I am running into issues where I get the following error:

Flow ID: 693e7e52-1fe3-4957-bc43-d95dc4af9fe7
Flow run ID: b8c1d85a-ef64-4c4d-a25d-386bd9f4d3b0
Flow run URL: <https://app.prefect.cloud/account/xxx/workspace/xxx/flow-runs/flow-run/xxx>
State message: Submission failed. prefect_aws.ecs.TaskFailedToStart: Timeout waiting for network interface provisioning to complete.

I can confirm that I see the same issue on the ecs cluster task ( image ) - I see this is caused by this Is there any way we could perhaps get some form of retry logic in the ECSTask infrastructure block? Is this something that should perhaps be a feature request? Thanks in advance

I appreciate this might be a fairly significant engineering effort, so not something I'm holding my breath for, but is there any appetite for Lambda-based task execution? We're probably heading for ECS tasks as a serverless execution environment, but they're kind of slow to spin up, and for short-lived tasks it feels like Lambda would be at the right sort of level

Hi #prefect-aws I am trying to execute my flow which is uploaded to s3 storage from a remote agent in a laptop in some other location. Getting this error. Any help would be great. Thanks

Flow could not be retrieved from deployment.
Traceback (most recent call last):
  File "C:\Users\91878\miniconda3\envs\scrapy\lib\site-packages\prefect\engine.py", line 255, in retrieve_flow_then_begin_flow_run
    flow = await load_flow_from_flow_run(flow_run, client=client)
  File "C:\Users\91878\miniconda3\envs\scrapy\lib\site-packages\prefect\client\utilities.py", line 47, in with_injected_client
    return await fn(*args, **kwargs)
  File "C:\Users\91878\miniconda3\envs\scrapy\lib\site-packages\prefect\deployments.py", line 157, in load_flow_from_flow_run
    storage_block = Block._from_block_document(storage_document)
  File "C:\Users\91878\miniconda3\envs\scrapy\lib\site-packages\prefect\blocks\core.py", line 540, in _from_block_document
    else cls.get_block_class_from_schema(block_document.block_schema)
  File "C:\Users\91878\miniconda3\envs\scrapy\lib\site-packages\prefect\blocks\core.py", line 571, in get_block_class_from_schema
    return lookup_type(cls, block_schema_to_key(schema))
  File "C:\Users\91878\miniconda3\envs\scrapy\lib\site-packages\prefect\utilities\dispatch.py", line 186, in lookup_type
    raise KeyError(
KeyError: "No class found for dispatch key 's3-bucket' in registry for type 'Block'."

Could anyone please explain why Prefect is attempting to register task definitions?

18:51:43.512 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'tasteful-ringtail': Registering task definition...

Is it possible to avoid this?

fyi -- aws us-east-2 region has a service health issue https://health.aws.amazon.com/health/status

does a deployment that uses an S3 storage block, download all the files from the object path before running the flow? or just the specified workflow file? I ask because we’re getting an ImportError from a workflow that’s like

import my_other_local_module

@flow
def foo():
  pass

Hey everyone! I'm not sure how to specify custom compute requirements via

infra_overrides

for a deployment that uses an

ECSTask

as its infrastructure block. I couldn't find any example that uses

Deployment.build_from_flow()

and

ECSTask

block. Also, I'm wondering whether specifying these overrides will register a new task definition at the time of flow run

Quick question: What happens when the ECS Service for an Agent does a restart? Assume that there are flow runs executing during this restart and there is only one agent and work queue

I’m getting an error in my agent logs, using an ECSTask Infrastructure Block

An error occured while monitoring flow run '793a0ee8-fe06-48d3-8858-31b2e99e4c50'. The flow run will not be marked as failed, but an issue may have occurred.

I think the Agent is trying to watch the created ECS Task’s logs, and using the wrong parameters to watch them

hello! I'm trying to define a waiter using prefect_aws.client_waiter, but it's failing because: Missing required parameter in input: "jobs" Traceback in the thread.

Any chance we can get a release of

prefect-aws

so that the recently merged PR is available to play around with ? 🙏

I have been working through creating an ECSTask as an infrastructure block for flow deployments and through extensive trial and error found some observations that were not part of the documentation: 1. If setting a

task_definition

,

containerDefinitions.name

must be “prefect”. Otherwise the new registered task definition will contain two separate

containerDefinitions

items and runs will fail. 2. I believe that the

command

parameter has to be blank/null, otherwise your flow will not be run properly. 3. If using the

task_customizations

parameter, you must cast your array(dict) as a

JsonPatch

object otherwise the pydantic validation will fail. This conflicts with the documentation and the examples in the

dataflow-ops

repo (@Anna Geller)

task_customizations=JsonPatch([
        {
            "op": "replace",
            "path": "/networkConfiguration/awsvpcConfiguration/assignPublicIp",
            "value": "DISABLED",
        },
    ])

Hey all, my team has been using ECSTask block infrastructure for a couple months, and we've noticed that flows will occasionally fail with an error similar to

Submission failed. RuntimeError: Timed out after 240.94525742530823s while watching task for status {until_status or 'STOPPED'}

, and when flows fail like this, it doesn't trigger an automatic retry, even if the flow has been configured to do so. Any ideas how I can get the retries to kick in so we don't get these occasional errors?

Hey team! We’re trying to setup some ECS based architecture that is effectively: 1. An ECS cluster which contains prefect plus a prefect aws agent 2. The above prefect instance configured with ECSTask block and aws blocks and s3 blocks (we’re new to blocks, trying to upgrade from prefect 1) using terraform (or some version of code) that will be instantiated in a gitlab build pipeline 3. The agent then runs tasks by spinning up a separate ECS. Eg for training ML models Has anyone come across (or made) some tutorials / guides that might be a good starting point for us? We’ve found https://towardsdatascience.com/prefect-aws-ecs-fargate-github-actions-make-serverless-dataflows-as-easy-as-py-f6025335effc#8d94 but its quite different (using prefect cloud and github actions)

Hello all, Despite reading the various posts about the "Access Denied" S3 error ; I can't get my deployment to upload to S3 from my local laptop 😞.

botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied

+

PermissionError: Access Denied

when doing

prefect deployment build flow-example.py:hi -q default -n flow-example-aws-d --apply -sb s3/aws-personal-s3

I am using a IAM user key/secret, it works well for uploading files via AWS CLI, and Prefect Cloud Blocks (named . As I'm discovering both Prefect Blocks and AWS S3/IAM management ; I have a few questions: • are there constraints regarding the S3 buckets ? (regions, public access ?) • do I need both AWS Credential and S3 blocks created in ? • do I need to use manually-created nested blocks ? (but I don't see the point) • do I need to use IAM roles (which I haven't studied yet 😇 ) boto* libs

- boto3=1.26.50
- botocore=1.29.50
- aiobotocore==2.4.2
- mypy-boto3==1.26.50
- mypy-boto3-s3==1.26.0.post1

prefect* version

prefect==2.7.8+4.ge291d4c18
prefect-aws==0.2.3

Currently running into this issue:

botocore.errorfactory.ClientException: An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family.

Our workflow is that we are using an

ECSTask

block with github storage.

prefect-aws==0.1.8
prefect==2.6.8

This is mainly caused when a mother flow fires off a subflow via

run_deployment

, but can occur when trying to invoke a standalone flow

Has anyone been able to set tags to be propagated from the task definition to tasks using ECS Task blocks?

Hi! I am just coming over from Prefect 1 to 2, and also thinking about how to set things up to run more efficiently. I'm having a hard time wrapping my head around some of the new concepts, hoping someone can point me in the right direction. I have a flow where some tasks require very little resources but can take a long time (e.g. trigger execution of a long-running procedure in a database), and others are very resource-heavy (data processing done in task's Python code). They would clearly benefit from running on different infrastructure. But as I understand it, each flow must run on one infrastructure? What is the right way to approach this kind of problem from a Prefect perspective?

I'm currently attempting to configure our flow run deployments such that they can override the

ECSTask

infra block by specifying custom values for

cpu

and

memory

. Any help is appreciated!

We're trying to get a POC going on Prefect 2. We have our infra setup, but are having some issues running flows on task definitions. It seems like this is happening: • First run of a flow gets the following error

prefect_aws.ecs.TaskFailedToStart: Cannotpullcontainererror: ref pull has been retried 1 time(s): failed to copy: httpReadSeeker: failed open: unexpected status code <https://registry-1.docker.io/v2/prefecthq/prefect/blobs/sha256:209b7d3b67f9dca78847c83ea6f0a78b0f91672ea1ccedc00e9bd5eb46315807>: 400 Bad Request

• 2nd and subsequent runs of the same flow gets a different error:

Flow could not be retrieved from deployment. Traceback (most recent call last): File "<frozen importlib._bootstrap_external>", line 883, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "/opt/prefect/flows/atlan.py", line 5, in <module> from util import SnowflakeDestination File "/opt/prefect/flows/util.py", line 3, in <module> import pandas as pd ModuleNotFoundError: No module named 'pandas' The above exception was the direct cause of the following exception: ScriptError: Script at 'flows/atlan.py' encountered an exception: ModuleNotFoundError("No module named 'pandas'")

• it appears that Prefect is creating a new task definition but it is not derived from our agent and therefore does not have the same dependencies installed. How can we resolve this?

Hey everyone! A few months ago, I posted a discussion on optimizing for high memory usage in the #best-practices-coordination-plane: https://prefect-community.slack.com/archives/C03D12VV4NN/p1671793942895089 We're still running into this issue. Idk if this apparent build up of memory usage is a bug with prefect or if it is something we can control