https://prefect.io
Join Slack
Hi there, looking for some help running an ECS Agent with S3 storage. Getting a weird boto error.
# ask-community
s
Hi there, looking for some help running an ECS Agent with S3 storage. Getting a weird boto error.
Copy code
import prefect
from prefect import task, Flow
from prefect.storage import S3

@task
def print_task():
    logger = prefect.context.get("logger")
    <http://logger.info|logger.info>(f'hello world')

with Flow("hello-flow") as flow:
    print_task()

flow.storage = S3(bucket="****")

if __name__ == '__main__':
    flow.run()
I was able to get an ECS Service running the agent. So I do have an agent running under the "Agents" tab in the Prefect Cloud UI.
I was able to get past 
Error downloading Flow from S3: Unable to locate credentials
by providing all the possible combinations of AWS env vars in the task definition container.
Now I'm getting something that looks like a boto error:
Copy code
15:58:12
INFO
agent
Submitted for execution: Task ****
15:58:47
INFO
S3
Downloading flow from s3://***/hello-flow/2022-03-29t04-10-25-834594-00-00
15:58:47
ERROR
S3
Error downloading Flow from S3: 'str' object has no attribute 'get'
15:58:48
ERROR
execute flow-run
Failed to load and execute flow run: AttributeError("'str' object has no attribute 'get'")
any help appreciated 🙏
(I'm running 
prefecthq/prefect:1.1.0-python3.8
as agent image)
a
@Sacha Ventura Here are some resources that may help you get started with Prefect and an ECS agent: 1. Blog post explaining how to set up an ECS agent as an ECS service Medium – 1 Nov 21 How to Cut Your AWS ECS Costs with Fargate Spot and Prefect Serverless Data Engineering Pipelines in Python Reading time: 11 min read 1. This repo has many examples with various storage and run_configuration options - check for those that contain 
ecs
in its name: https://github.com/anna-geller/packaging-prefect-flows/tree/master/flows
The AttributeError you get comes from boto3, not Prefect. you need to create a task role (IAM role) with S3 permissions, ECS won't work with credentials provided as env variables
the blog post above shows how you can do that
s
Amazing @Anna Geller thank you for your answers! I’ll give it a go
🙌 1
Everything worked from the first go - @Anna Geller thank you so much
🙌 1
21 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by