https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-aws
  • b

    Ben Muller

    11/16/2022, 11:31 PM
    Hey, any idea when there will be a new official release for prefect-aws? I'm waiting on the handling of the rate limits for ecs. I know I can just install from the repo, but was hoping it'd be official soon.
    🙌 1
    ✅ 1
    a
    a
    2 replies ¡ 3 participants
  • f

    Fred Birchall

    11/17/2022, 9:55 AM
    Hi All! I’m wondering if the
    prefect-aws
    block has any future scope to handle Lambda invokes, more specifically long running functions (over 300 seconds). The issue that my team is hitting is due to
    TCP keepalive
    . We are self hosting our prefect instance using AWS EKS and seen that with some k8s magic (when using EC2 instances) you can redefine the system configuration to send the first TCP keepalive message after say 200 seconds instead of the Linux default of 2 hours, but this solution does not work using AWS Fargate (our current set-up). So I was wondering if this issue is shared with others and if there is a potential solution down the line? Thanks!
  • a

    Anna Geller

    11/17/2022, 1:29 PM
    new prefect-aws release
    :party-parrot: 3
    :aws: 1
  • в

    Владислав Богучаров

    11/17/2022, 2:14 PM
    Hi! :blob-attention-gif: First of all, thank you for the excellent template on integrating Prefect with AWS. As I understand it, it is now possible to run all ECS tasks only with a specific CPU and Memory, which we prescribe in the ECS Block. But is it possible to configure the CPU and Memory for each task separately? Because for some tasks I need min capacities, for others I need maximum ones.
    a
    1 reply ¡ 2 participants
  • v

    Vishnu Duggirala

    11/17/2022, 4:30 PM
    This might be a dumb question, Is it possible to deploy my image as an ECS task without using infrastructure blocks?
    a
    14 replies ¡ 2 participants
  • c

    Claire Herdeman

    11/17/2022, 7:07 PM
    Ok, hopefully my last question for ironing out our setup, as of this morning I'm seeing the error in the thread when deploying a flow
    ✅ 1
    m
    15 replies ¡ 2 participants
  • j

    James Zhang

    11/21/2022, 2:40 PM
    hey guys, I’m trying to
    prefect deployment build
    with
    -sb
    on S3 and
    -ib
    of a DockerContainer, then I got
    botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
    I guess it has to do with the IAM role, but I don’ know how to switch/use a IAM role during
    deployment build
    , any ideas?
    ✅ 1
    a
    2 replies ¡ 2 participants
  • v

    Vishnu Duggirala

    11/22/2022, 8:43 PM
    Is anyone else experiencing this issue where it does not delete task-def, even when
    auto_deregister_task_definition
    set to True? Is it normal for a deployment to create a new task-definition every time it runs an ECS task?
    ✅ 1
    a
    m
    6 replies ¡ 3 participants
  • i

    Ikkyu Choi

    11/22/2022, 11:55 PM
    Hi, I’m trying to sending alarm to slack when my flow failed. Normal case it works well, but when flow about lack of aws resource (i.e., gpu) fails i didnt’t get the alarm. Now I’m using ECS agnet of prefect 0.15.3. Anyone could help?
    m
    1 reply ¡ 2 participants
  • s

    Santiago Gonzalez

    11/25/2022, 6:31 PM
    Hi, This is kind of difficult to explain. I have a flow that performs a job over an EC2 Instance, and the problem is that sometimes it fails and the most of times, it succeed. The job that runs, do the following operations: 1- Download a jar from somewhere. 2- Download some script from github. 3- Execute the script with some sort of arguments. And what it does, is download data from AWS S3, process the files downloaded, and then upload the results. The errors that I usually got are like •
    Main class from jar could not be found
    •
    The output directory does not exists, so it could not be synchronize to AWS S3
    Do you have any idea of why these types of issues happens time to time? BTW: I am using
    boto3
    SSM
    agent to handle EC2 Instances creation, execution and termination
    a
    3 replies ¡ 2 participants
  • b

    Ben Muller

    11/29/2022, 9:18 PM
    Hey prefect-aws, I am running into issues where I get the following error:
    Flow ID: 693e7e52-1fe3-4957-bc43-d95dc4af9fe7
    Flow run ID: b8c1d85a-ef64-4c4d-a25d-386bd9f4d3b0
    Flow run URL: <https://app.prefect.cloud/account/xxx/workspace/xxx/flow-runs/flow-run/xxx>
    State message: Submission failed. prefect_aws.ecs.TaskFailedToStart: Timeout waiting for network interface provisioning to complete.
    I can confirm that I see the same issue on the ecs cluster task ( image ) - I see this is caused by this Is there any way we could perhaps get some form of retry logic in the ECSTask infrastructure block? Is this something that should perhaps be a feature request? Thanks in advance
    ✅ 1
    a
    6 replies ¡ 2 participants
  • c

    Christopher

    11/30/2022, 2:38 PM
    I appreciate this might be a fairly significant engineering effort, so not something I'm holding my breath for, but is there any appetite for Lambda-based task execution? We're probably heading for ECS tasks as a serverless execution environment, but they're kind of slow to spin up, and for short-lived tasks it feels like Lambda would be at the right sort of level
    ✅ 1
    a
    m
    +2
    26 replies ¡ 5 participants
  • a

    ashish karan

    12/01/2022, 1:25 PM
    Hi #prefect-aws I am trying to execute my flow which is uploaded to s3 storage from a remote agent in a laptop in some other location. Getting this error. Any help would be great. Thanks
    Flow could not be retrieved from deployment.
    Traceback (most recent call last):
      File "C:\Users\91878\miniconda3\envs\scrapy\lib\site-packages\prefect\engine.py", line 255, in retrieve_flow_then_begin_flow_run
        flow = await load_flow_from_flow_run(flow_run, client=client)
      File "C:\Users\91878\miniconda3\envs\scrapy\lib\site-packages\prefect\client\utilities.py", line 47, in with_injected_client
        return await fn(*args, **kwargs)
      File "C:\Users\91878\miniconda3\envs\scrapy\lib\site-packages\prefect\deployments.py", line 157, in load_flow_from_flow_run
        storage_block = Block._from_block_document(storage_document)
      File "C:\Users\91878\miniconda3\envs\scrapy\lib\site-packages\prefect\blocks\core.py", line 540, in _from_block_document
        else cls.get_block_class_from_schema(block_document.block_schema)
      File "C:\Users\91878\miniconda3\envs\scrapy\lib\site-packages\prefect\blocks\core.py", line 571, in get_block_class_from_schema
        return lookup_type(cls, block_schema_to_key(schema))
      File "C:\Users\91878\miniconda3\envs\scrapy\lib\site-packages\prefect\utilities\dispatch.py", line 186, in lookup_type
        raise KeyError(
    KeyError: "No class found for dispatch key 's3-bucket' in registry for type 'Block'."
    ✅ 1
    a
    r
    4 replies ¡ 3 participants
  • v

    Vishnu Duggirala

    12/02/2022, 8:46 PM
    Could anyone please explain why Prefect is attempting to register task definitions?
    18:51:43.512 | INFO    | prefect.infrastructure.ecs-task - ECSTask 'tasteful-ringtail': Registering task definition...
    Is it possible to avoid this?
    ✅ 1
    a
    m
    +1
    11 replies ¡ 4 participants
  • k

    Kalise Richmond

    12/05/2022, 8:54 PM
    fyi -- aws us-east-2 region has a service health issue https://health.aws.amazon.com/health/status
    ✅ 2
    1 reply ¡ 1 participant
  • a

    Alex Turek

    12/20/2022, 6:28 PM
    does a deployment that uses an S3 storage block, download all the files from the object path before running the flow? or just the specified workflow file? I ask because we’re getting an ImportError from a workflow that’s like
    import my_other_local_module
    
    @flow
    def foo():
      pass
    3 replies ¡ 1 participant
  • k

    Kelvin DeCosta

    12/21/2022, 3:05 PM
    Hey everyone! I'm not sure how to specify custom compute requirements via
    infra_overrides
    for a deployment that uses an
    ECSTask
    as its infrastructure block. I couldn't find any example that uses
    Deployment.build_from_flow()
    and
    ECSTask
    block. Also, I'm wondering whether specifying these overrides will register a new task definition at the time of flow run
    ✅ 1
    a
    1 reply ¡ 2 participants
  • k

    Kelvin DeCosta

    12/23/2022, 9:02 AM
    Quick question: What happens when the ECS Service for an Agent does a restart? Assume that there are flow runs executing during this restart and there is only one agent and work queue
    ✅ 1
    b
    4 replies ¡ 2 participants
  • a

    Alex Turek

    12/30/2022, 9:51 PM
    I’m getting an error in my agent logs, using an ECSTask Infrastructure Block
    An error occured while monitoring flow run '793a0ee8-fe06-48d3-8858-31b2e99e4c50'. The flow run will not be marked as failed, but an issue may have occurred.
    I think the Agent is trying to watch the created ECS Task’s logs, and using the wrong parameters to watch them
    a
    m
    8 replies ¡ 3 participants
  • g

    Gabriela Palacios

    01/03/2023, 9:16 PM
    hello! I'm trying to define a waiter using prefect_aws.client_waiter, but it's failing because: Missing required parameter in input: "jobs" Traceback in the thread.
    2 replies ¡ 1 participant
  • b

    Ben Muller

    01/04/2023, 10:08 PM
    Any chance we can get a release of
    prefect-aws
    so that the recently merged PR is available to play around with ? 🙏
    👀 1
    a
    k
    34 replies ¡ 3 participants
  • l

    Lawrence Lee

    01/06/2023, 8:55 AM
    I have been working through creating an ECSTask as an infrastructure block for flow deployments and through extensive trial and error found some observations that were not part of the documentation: 1. If setting a
    task_definition
    ,
    containerDefinitions.name
    must be “prefect”. Otherwise the new registered task definition will contain two separate
    containerDefinitions
    items and runs will fail. 2. I believe that the
    command
    parameter has to be blank/null, otherwise your flow will not be run properly. 3. If using the
    task_customizations
    parameter, you must cast your array(dict) as a
    JsonPatch
    object otherwise the pydantic validation will fail. This conflicts with the documentation and the examples in the
    dataflow-ops
    repo (@Anna Geller)
    task_customizations=JsonPatch([
            {
                "op": "replace",
                "path": "/networkConfiguration/awsvpcConfiguration/assignPublicIp",
                "value": "DISABLED",
            },
        ])
    m
    a
    6 replies ¡ 3 participants
  • m

    Mike Grabbe

    01/10/2023, 3:33 PM
    Hey all, my team has been using ECSTask block infrastructure for a couple months, and we've noticed that flows will occasionally fail with an error similar to
    Submission failed. RuntimeError: Timed out after 240.94525742530823s while watching task for status {until_status or 'STOPPED'}
    , and when flows fail like this, it doesn't trigger an automatic retry, even if the flow has been configured to do so. Any ideas how I can get the retries to kick in so we don't get these occasional errors?
    ✅ 1
    k
    m
    11 replies ¡ 3 participants
  • s

    Samuel Hinton

    01/12/2023, 2:57 AM
    Hey team! We’re trying to setup some ECS based architecture that is effectively: 1. An ECS cluster which contains prefect plus a prefect aws agent 2. The above prefect instance configured with ECSTask block and aws blocks and s3 blocks (we’re new to blocks, trying to upgrade from prefect 1) using terraform (or some version of code) that will be instantiated in a gitlab build pipeline 3. The agent then runs tasks by spinning up a separate ECS. Eg for training ML models Has anyone come across (or made) some tutorials / guides that might be a good starting point for us? We’ve found https://towardsdatascience.com/prefect-aws-ecs-fargate-github-actions-make-serverless-dataflows-as-easy-as-py-f6025335effc#8d94 but its quite different (using prefect cloud and github actions)
  • c

    ClĂŠment Peyrard

    01/15/2023, 1:50 PM
    Hello all, Despite reading the various posts about the "Access Denied" S3 error ; I can't get my deployment to upload to S3 from my local laptop 😞.
    botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the PutObject operation: Access Denied
    +
    PermissionError: Access Denied
    when doing
    prefect deployment build flow-example.py:hi -q default -n flow-example-aws-d --apply -sb s3/aws-personal-s3
    I am using a IAM user key/secret, it works well for uploading files via AWS CLI, and Prefect Cloud Blocks (named . As I'm discovering both Prefect Blocks and AWS S3/IAM management ; I have a few questions: • are there constraints regarding the S3 buckets ? (regions, public access ?) • do I need both AWS Credential and S3 blocks created in ? • do I need to use manually-created nested blocks ? (but I don't see the point) • do I need to use IAM roles (which I haven't studied yet 😇 ) boto* libs
    - boto3=1.26.50
    - botocore=1.29.50
    - aiobotocore==2.4.2
    - mypy-boto3==1.26.50
    - mypy-boto3-s3==1.26.0.post1
    prefect* version
    prefect==2.7.8+4.ge291d4c18
    prefect-aws==0.2.3
    m
    n
    11 replies ¡ 3 participants
  • j

    Joshua Grant

    01/25/2023, 4:33 PM
    Currently running into this issue:
    botocore.errorfactory.ClientException: An error occurred (ClientException) when calling the RegisterTaskDefinition operation: Too many concurrent attempts to create a new revision of the specified family.
    Our workflow is that we are using an
    ECSTask
    block with github storage.
    prefect-aws==0.1.8
    prefect==2.6.8
    This is mainly caused when a mother flow fires off a subflow via
    run_deployment
    , but can occur when trying to invoke a standalone flow
    b
    a
    15 replies ¡ 3 participants
  • k

    Kevin B

    01/30/2023, 7:20 PM
    Has anyone been able to set tags to be propagated from the task definition to tasks using ECS Task blocks?
    ✅ 1
    1 reply ¡ 1 participant
  • m

    Maciej

    02/03/2023, 9:02 PM
    Hi! I am just coming over from Prefect 1 to 2, and also thinking about how to set things up to run more efficiently. I'm having a hard time wrapping my head around some of the new concepts, hoping someone can point me in the right direction. I have a flow where some tasks require very little resources but can take a long time (e.g. trigger execution of a long-running procedure in a database), and others are very resource-heavy (data processing done in task's Python code). They would clearly benefit from running on different infrastructure. But as I understand it, each flow must run on one infrastructure? What is the right way to approach this kind of problem from a Prefect perspective?
    t
    5 replies ¡ 2 participants
  • k

    Kelvin DeCosta

    02/07/2023, 12:06 PM
    I'm currently attempting to configure our flow run deployments such that they can override the
    ECSTask
    infra block by specifying custom values for
    cpu
    and
    memory
    . Any help is appreciated!
    c
    s
    8 replies ¡ 3 participants
  • s

    Stephen Lloyd

    02/08/2023, 10:59 AM
    We're trying to get a POC going on Prefect 2. We have our infra setup, but are having some issues running flows on task definitions. It seems like this is happening: • First run of a flow gets the following error
    prefect_aws.ecs.TaskFailedToStart: Cannotpullcontainererror: ref pull has been retried 1 time(s): failed to copy: httpReadSeeker: failed open: unexpected status code <https://registry-1.docker.io/v2/prefecthq/prefect/blobs/sha256:209b7d3b67f9dca78847c83ea6f0a78b0f91672ea1ccedc00e9bd5eb46315807>: 400 Bad Request
    • 2nd and subsequent runs of the same flow gets a different error:
    Flow could not be retrieved from deployment. Traceback (most recent call last): File "<frozen importlib._bootstrap_external>", line 883, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "/opt/prefect/flows/atlan.py", line 5, in <module> from util import SnowflakeDestination File "/opt/prefect/flows/util.py", line 3, in <module> import pandas as pd ModuleNotFoundError: No module named 'pandas' The above exception was the direct cause of the following exception: ScriptError: Script at 'flows/atlan.py' encountered an exception: ModuleNotFoundError("No module named 'pandas'")
    • it appears that Prefect is creating a new task definition but it is not derived from our agent and therefore does not have the same dependencies installed. How can we resolve this?
    c
    2 replies ¡ 2 participants
Powered by Linen
Title
s

Stephen Lloyd

02/08/2023, 10:59 AM
We're trying to get a POC going on Prefect 2. We have our infra setup, but are having some issues running flows on task definitions. It seems like this is happening: • First run of a flow gets the following error
prefect_aws.ecs.TaskFailedToStart: Cannotpullcontainererror: ref pull has been retried 1 time(s): failed to copy: httpReadSeeker: failed open: unexpected status code <https://registry-1.docker.io/v2/prefecthq/prefect/blobs/sha256:209b7d3b67f9dca78847c83ea6f0a78b0f91672ea1ccedc00e9bd5eb46315807>: 400 Bad Request
• 2nd and subsequent runs of the same flow gets a different error:
Flow could not be retrieved from deployment. Traceback (most recent call last): File "<frozen importlib._bootstrap_external>", line 883, in exec_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed File "/opt/prefect/flows/atlan.py", line 5, in <module> from util import SnowflakeDestination File "/opt/prefect/flows/util.py", line 3, in <module> import pandas as pd ModuleNotFoundError: No module named 'pandas' The above exception was the direct cause of the following exception: ScriptError: Script at 'flows/atlan.py' encountered an exception: ModuleNotFoundError("No module named 'pandas'")
• it appears that Prefect is creating a new task definition but it is not derived from our agent and therefore does not have the same dependencies installed. How can we resolve this?
c

Claire Herdeman

02/08/2023, 3:48 PM
Could you clarify what you mean by "derived from our agent"?
s

Stephen Lloyd

02/09/2023, 9:09 AM
Thanks for clarifying. I meant derived from our agent's image. I got an answer from support@prefect.io on this last night. If I understand it correctly, anytime a block's Task definition parameters differ from the agent's task definition parameters Prefect spins up a separate task definition. This then means that the new task definition may not have the needed dependencies. Did I get that right?
View count: 1