Sacha Ventura

    Sacha Ventura

    6 months ago
    Hi there, looking for some help running an ECS Agent with S3 storage. Getting a weird boto error.
    import prefect
    from prefect import task, Flow
    from import S3
    def print_task():
        logger = prefect.context.get("logger")
        <|>(f'hello world')
    with Flow("hello-flow") as flow:
        print_task() = S3(bucket="****")
    if __name__ == '__main__':
    I was able to get an ECS Service running the agent. So I do have an agent running under the "Agents" tab in the Prefect Cloud UI.
    I was able to get past
    Error downloading Flow from S3: Unable to locate credentials
    by providing all the possible combinations of AWS env vars in the task definition container.
    Now I'm getting something that looks like a boto error:
    Submitted for execution: Task ****
    Downloading flow from s3://***/hello-flow/2022-03-29t04-10-25-834594-00-00
    Error downloading Flow from S3: 'str' object has no attribute 'get'
    execute flow-run
    Failed to load and execute flow run: AttributeError("'str' object has no attribute 'get'")
    any help appreciated 🙏
    (I'm running
    as agent image)
    Anna Geller

    Anna Geller

    6 months ago
    @Sacha Ventura Here are some resources that may help you get started with Prefect and an ECS agent: 1. Blog post explaining how to set up an ECS agent as an ECS service Medium – 1 Nov 21 How to Cut Your AWS ECS Costs with Fargate Spot and Prefect Serverless Data Engineering Pipelines in Python Reading time: 11 min read1. This repo has many examples with various storage and run_configuration options - check for those that contain
    in its name:
    The AttributeError you get comes from boto3, not Prefect. you need to create a task role (IAM role) with S3 permissions, ECS won't work with credentials provided as env variables
    the blog post above shows how you can do that
    Sacha Ventura

    Sacha Ventura

    5 months ago
    Amazing @Anna Geller thank you for your answers! I’ll give it a go
    Everything worked from the first go - @Anna Geller thank you so much