Thread
#prefect-community
    Sacha Ventura

    Sacha Ventura

    6 months ago
    Hi there, looking for some help running an ECS Agent with S3 storage. Getting a weird boto error.
    import prefect
    from prefect import task, Flow
    from prefect.storage import S3
    
    @task
    def print_task():
        logger = prefect.context.get("logger")
        <http://logger.info|logger.info>(f'hello world')
    
    with Flow("hello-flow") as flow:
        print_task()
    
    flow.storage = S3(bucket="****")
    
    if __name__ == '__main__':
        flow.run()
    I was able to get an ECS Service running the agent. So I do have an agent running under the "Agents" tab in the Prefect Cloud UI.
    I was able to get past
    Error downloading Flow from S3: Unable to locate credentials
    by providing all the possible combinations of AWS env vars in the task definition container.
    Now I'm getting something that looks like a boto error:
    15:58:12
    INFO
    agent
    Submitted for execution: Task ****
    15:58:47
    INFO
    S3
    Downloading flow from s3://***/hello-flow/2022-03-29t04-10-25-834594-00-00
    15:58:47
    ERROR
    S3
    Error downloading Flow from S3: 'str' object has no attribute 'get'
    15:58:48
    ERROR
    execute flow-run
    Failed to load and execute flow run: AttributeError("'str' object has no attribute 'get'")
    any help appreciated 🙏
    (I'm running
    prefecthq/prefect:1.1.0-python3.8
    as agent image)
    Anna Geller

    Anna Geller

    6 months ago
    @Sacha Ventura Here are some resources that may help you get started with Prefect and an ECS agent: 1. Blog post explaining how to set up an ECS agent as an ECS service Medium – 1 Nov 21 How to Cut Your AWS ECS Costs with Fargate Spot and Prefect Serverless Data Engineering Pipelines in Python Reading time: 11 min read1. This repo has many examples with various storage and run_configuration options - check for those that contain
    ecs
    in its name:https://github.com/anna-geller/packaging-prefect-flows/tree/master/flows
    The AttributeError you get comes from boto3, not Prefect. you need to create a task role (IAM role) with S3 permissions, ECS won't work with credentials provided as env variables
    the blog post above shows how you can do that
    Sacha Ventura

    Sacha Ventura

    5 months ago
    Amazing @Anna Geller thank you for your answers! I’ll give it a go
    Everything worked from the first go - @Anna Geller thank you so much