https://prefect.io logo
Powered by
Title
r

Ritabrata Moitra

05/25/2023, 1:03 PM
Hey folks. I am facing issues trying to retrieve the flow code from my remote storage S3. The deployment submission goes through, but then fails with the following simplified stacktrace - 
12:30:23.877 | INFO    | Flow run 'refreshing-coyote' - Downloading flow code from storage at ''
12:30:36.187 | ERROR   | Flow run 'refreshing-coyote' - Flow could not be retrieved from deployment.
....
socket.gaierror: [Errno -5] No address associated with hostname
...
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host <bucket-name>.<http://s3.eu-west-1.amazonaws.com:443|s3.eu-west-1.amazonaws.com:443> ssl:default [No address associated with hostname]
...
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://<bucket-name>.<http://s3.eu-west-1.amazonaws.com/?list-type=2&prefix=&delimiter=&encoding-type=url|s3.eu-west-1.amazonaws.com/?list-type=2&prefix=&delimiter=&encoding-type=url>"
...
I am using a ECS task infra block ( definition in comments ) and building my deployment like this - 
prefect deployment build ecr_flow.py:ecr_flow -n ecr_flow_path -sb s3/prefect-s3-storage --path opt/prefect/flows -a -q rito
Any pointers where I could be going wrong?
✅ 1
My task definition - 
from prefect import flow
from prefect_aws import AwsCredentials
from prefect_aws.ecs import ECSTask


@flow
def ecr_flow():
    ecr_task = ECSTask(
        image="<http://xxx.dkr.ecr.eu-west-1.amazonaws.com/test-prefect:latest|xxx.dkr.ecr.eu-west-1.amazonaws.com/test-prefect:latest>",
        credentials=AwsCredentials.load("xxx"),
        cluster="arn:aws:ecs:eu-west-1:xxx:cluster/prefect-agent-cluster-2",
        stream_output=True,
        configure_cloudwatch_logs=True,
        region="eu-west-1",
        vpc_id="vpc-xxx",
        command=["python3", "/opt/prefect/flows/sample_deployment.py"],
        execution_role_arn="arn:aws:iam::xxx:role/test-iam-role-prefect",
        task_role_arn="arn:aws:iam::xxx:role/test-iam-role-prefect",
        auto_deregister_task_definition=True
    )
    return ecr_task.run()


if __name__ == '__main__':
    ecr_flow()
c

Christopher Boyd

05/25/2023, 1:52 PM
This looks like a network / DNS issue - 
socket.gaierror: [Errno -5] No address associated with hostname
...
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host <bucket-name>.<http://s3.eu-west-1.amazonaws.com:443|s3.eu-west-1.amazonaws.com:443> ssl:default [No address associated with hostname]
Where is this running from, and can you resolve that address manually?
r

Ritabrata Moitra

05/25/2023, 1:54 PM
Thanks for the response Christopher. Indeed, looks like a DNS issue. What I am failing to understand is why does is boto trying to hit an endpoint, instead of using 
s3fs
to download the flow My infra config looks like this - 1. Server - EKS Cluster 2. Agent - Local docker container ( has the AWS credentials volume mounted ) 3. ECS Task I actually can’t resolve that address manually. Seems to be something malformed.
c

Christopher Boyd

05/25/2023, 1:56 PM
s3fs is built on top of botocore - https://aiobotocore.readthedocs.io/en/latest/
https://pypi.org/project/s3fs/
If you are running infra in EKS and ECS, I would expect you to have an agent in those areas as well
since those would be the agents submitting the work to their respective infrastructures
Where is that stack trace from?
r

Ritabrata Moitra

05/25/2023, 1:59 PM
This is from my docker container, which runs the agent.. Would local agents not be able to submit flows to ECS infra?
c

Christopher Boyd

05/25/2023, 1:59 PM
Your local agent doesn’t have any permissions to submit or execute , and no DNS resolution it seems?
If you’re running work in ECS, I’d expect to see an agent running in ECS like https://github.com/PrefectHQ/prefect-recipes/tree/main/devops/infrastructure-as-code/aws/cli-prefect2-ecs-agent or https://github.com/PrefectHQ/prefect-recipes/tree/main/devops/infrastructure-as-code/aws/tf-prefect2-ecs-agent
You can check out some guides here for reference
r

Ritabrata Moitra

05/25/2023, 2:11 PM
Okay thanks a ton for the pointers Christopher. Let me try that out 🙌
#prefect-communityJoin Slack
Powered by