https://prefect.io logo
a

ale

10/19/2020, 11:57 AM
Hi folks, today I’m facing a strange issue. I’m using
AWSSecretsManager
in a flow which runs on Prefect Fargate Agent. The flow works fine if the flow runs with
launch_type="FARGATE"
If the flow runs with
launch_type="EC2"
then I get the following error:
Copy code
botocore.exceptions.NoRegionError: You must specify a region.
It seems that the flow cannot pick the region from the provided task/execution role… Any suggestions?
r

Raphaël Riel

10/19/2020, 12:23 PM
I have no definitive answer to this, but from what I understand, the SecretID is the same across both LaunchType. Since one of the two doesn’t work, I assume the SecretID is not using the ARN-type (Which would contains the Region code). This would led me to think that somehow the Region in FarGate is given as an ENV Var, and that it is not the case in EC2. (ENV var that Boto3 would recover at runtime) Hope this help you figuring out your problem!
What is the format your are using for the SecretID?
a

ale

10/19/2020, 12:28 PM
Hi @Raphaël Riel! The secret ID is passed in using the ARN-type
An interesting thing I’ve found is that setting the env var
AWS_DEFAULT_REGION
makes AWSSecretManager tasks to run indefinitely (it remains in the Submitted status forever)
r

Raphaël Riel

10/19/2020, 12:31 PM
Oh, so the problem lies in booting an Agent on AWS, and not retrieving the secret!
Are you able to confirm in ECS Console that the Task is being started and that there is capacity in your Cluster?
a

ale

10/19/2020, 12:40 PM
Yep, I can confirm
r

Raphaël Riel

10/19/2020, 12:51 PM
Are the EC2's underlying VPC/Networks allowing outbound connections to the internet? If your EC2-based container can’t communicate back to the PRefect’s server or cloud, it wont be able to pick up flows.
(Or report as an agent being ready)
a

ale

10/19/2020, 12:53 PM
Other flows are running fine so it seems not to be a problem related to networking…
r

Raphaël Riel

10/19/2020, 12:55 PM
Ok. Unfortunately I’m out of ideas 😛 I wish you good luck with that!
👍 1
s

Spencer

10/19/2020, 1:02 PM
It's the Fargate Agent, I don't think it's intended to run on EC2 at all
a

ale

10/19/2020, 1:26 PM
@josh said that the Fargate Agent should be able to run flows on both Fargate and EC2 https://prefect-community.slack.com/archives/C014Z8DPDSR/p1602767211198500?thread_ts=1602751641.197400&cid=C014Z8DPDSR
👍 1
Maybe @josh can help 🙂
Seems to be a network issue. The container in the EC2 network does not have internet access. We tried to use networkMode
bridge
, but then we realised that Fargate Agent set it to
awsvpc
when creating new tasks…
I'm going to open a PR to support bridge networkMode. Do you guys see any issues with this mode compared with awsvpc?
s

Spencer

10/20/2020, 12:43 PM
Is this the task to start the flow or the flow run itself? ¯\_(ツ)_/¯
9 Views