Hey folks. I am facing issues trying to retrieve t...
# ask-communityr
Ritabrata Moitra
05/25/2023, 1:03 PMHey folks. I am facing issues trying to retrieve the flow code from my remote storage S3. The deployment submission goes through, but then fails with the following simplified stacktrace -
Copy code
12:30:23.877 | INFO | Flow run 'refreshing-coyote' - Downloading flow code from storage at ''
12:30:36.187 | ERROR | Flow run 'refreshing-coyote' - Flow could not be retrieved from deployment.
....
socket.gaierror: [Errno -5] No address associated with hostname
...
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host <bucket-name>.<http://s3.eu-west-1.amazonaws.com:443|s3.eu-west-1.amazonaws.com:443> ssl:default [No address associated with hostname]
...
botocore.exceptions.EndpointConnectionError: Could not connect to the endpoint URL: "https://<bucket-name>.<http://s3.eu-west-1.amazonaws.com/?list-type=2&prefix=&delimiter=&encoding-type=url|s3.eu-west-1.amazonaws.com/?list-type=2&prefix=&delimiter=&encoding-type=url>"
...prefect deployment build ecr_flow.py:ecr_flow -n ecr_flow_path -sb s3/prefect-s3-storage --path opt/prefect/flows -a -q rito✅ 1
Ritabrata Moitra
05/25/2023, 1:04 PMMy task definition -
Copy code
from prefect import flow
from prefect_aws import AwsCredentials
from prefect_aws.ecs import ECSTask
@flow
def ecr_flow():
ecr_task = ECSTask(
image="<http://xxx.dkr.ecr.eu-west-1.amazonaws.com/test-prefect:latest|xxx.dkr.ecr.eu-west-1.amazonaws.com/test-prefect:latest>",
credentials=AwsCredentials.load("xxx"),
cluster="arn:aws:ecs:eu-west-1:xxx:cluster/prefect-agent-cluster-2",
stream_output=True,
configure_cloudwatch_logs=True,
region="eu-west-1",
vpc_id="vpc-xxx",
command=["python3", "/opt/prefect/flows/sample_deployment.py"],
execution_role_arn="arn:aws:iam::xxx:role/test-iam-role-prefect",
task_role_arn="arn:aws:iam::xxx:role/test-iam-role-prefect",
auto_deregister_task_definition=True
)
return ecr_task.run()
if __name__ == '__main__':
ecr_flow()c
Christopher Boyd
05/25/2023, 1:52 PMThis looks like a network / DNS issue -
Copy code
socket.gaierror: [Errno -5] No address associated with hostname
...
aiohttp.client_exceptions.ClientConnectorError: Cannot connect to host <bucket-name>.<http://s3.eu-west-1.amazonaws.com:443|s3.eu-west-1.amazonaws.com:443> ssl:default [No address associated with hostname]Christopher Boyd
05/25/2023, 1:52 PMWhere is this running from, and can you resolve that address manually?
r
Ritabrata Moitra
05/25/2023, 1:54 PMThanks for the response Christopher. Indeed, looks like a DNS issue. What I am failing to understand is why does is boto trying to hit an endpoint, instead of using
s3fsc
Christopher Boyd
05/25/2023, 1:56 PMs3fs is built on top of botocore - https://aiobotocore.readthedocs.io/en/latest/
Christopher Boyd
05/25/2023, 1:56 PMChristopher Boyd
05/25/2023, 1:57 PMIf you are running infra in EKS and ECS, I would expect you to have an agent in those areas as well
Christopher Boyd
05/25/2023, 1:57 PMsince those would be the agents submitting the work to their respective infrastructures
Christopher Boyd
05/25/2023, 1:57 PMWhere is that stack trace from?
r
Ritabrata Moitra
05/25/2023, 1:59 PMThis is from my docker container, which runs the agent..
Would local agents not be able to submit flows to ECS infra?
c
Christopher Boyd
05/25/2023, 1:59 PMYour local agent doesn’t have any permissions to submit or execute , and no DNS resolution it seems?
Christopher Boyd
05/25/2023, 2:00 PMIf you’re running work in ECS, I’d expect to see an agent running in ECS like https://github.com/PrefectHQ/prefect-recipes/tree/main/devops/infrastructure-as-code/aws/cli-prefect2-ecs-agent or https://github.com/PrefectHQ/prefect-recipes/tree/main/devops/infrastructure-as-code/aws/tf-prefect2-ecs-agent
Christopher Boyd
05/25/2023, 2:01 PMYou can check out some guides here for reference
r
Ritabrata Moitra
05/25/2023, 2:11 PMOkay thanks a ton for the pointers Christopher. Let me try that out 🙌
6 Views