Darragh
07/14/2020, 8:27 AMFailed to load and execute Flow's environment: ClientError('An error occurred (UnrecognizedClientException) when calling the RegisterTaskDefinition operation: The security token included in the request is invalid')
Sample config for the environment:
flow.environment = FargateTaskEnvironment(
launch_type="FARGATE",
region="us-east-1",
aws_session_token="FwoGZXIvYXdzEOL//////////wEaDFSgRMmr3M07yXJ3gCKCAdZO6f/LRZc6b7DjDip0lrTvCa+FDQpFAGJEyB6Ka1tF9By3fTgKkbqSM6EnuHQgTEviQJrOn13E7wlvKKVV1++YGaa3gb1Pn9q12BxCN7I6SvQ8oBW9AE73Judo0tuYTdTc5eYC7m2PaYU/d5fkRIj29EJWp9EpO3+yq/S1saxiGvYosKGy+AUyKBOOerH4ymGN9lxo/5aprU5DXumyaC6yg2satDNNoUdPaSBQ2R8fNq0=",
cpu="256",
memory="512",
enable_task_revisions="True",
....
....
)
Zachary Hughes
07/14/2020, 1:07 PMDarragh
07/14/2020, 1:47 PMZachary Hughes
07/14/2020, 2:22 PMDarragh
07/14/2020, 2:42 PMZachary Hughes
07/14/2020, 3:09 PMDarragh
07/14/2020, 4:32 PMFailed to load and execute Flow's environment: ParamValidationError('Parameter validation failed:\nMissing required parameter in input: "taskDefinition"')
Zachary Hughes
07/14/2020, 5:04 PMaws_session_token
. We can definitely open an issue for that change if you'd like.Darragh
07/14/2020, 5:08 PMaws_session_token
is definitely needed?
From the output I got above [running without the token] my config is definitely missing something, it’s just not telling me what it is 🙂Zachary Hughes
07/14/2020, 5:10 PMDarragh
07/14/2020, 5:21 PMMissing required parameter
- it doesn’t seem to be specific to the token - is there any way of getting more info on what that error is?Zachary Hughes
07/14/2020, 5:21 PMtaskDefinition
. I think you'll also need to pass a taskDefinition
that matches the family
, as mentioned in this link:
https://docs.prefect.io/api/latest/environments/execution.html#fargatetaskenvironmentMissing required parameter
is coming from boto3 itself, but taking a look to see if we can find any additional information.family
argument from the task_definition_name
, so that could definitely be the missing parameter for taskDefinition
.Darragh
07/14/2020, 5:41 PMFailed to load and execute Flow's environment: InvalidParameterException('An error occurred (InvalidParameterException) when calling the RunTask operation: Task definition does not support launch_type FARGATE.')
Zachary Hughes
07/14/2020, 6:55 PMlaunch_type
is FARGATE-- do you see this error even when you don't specify launch_type
in your environment?Darragh
07/14/2020, 7:03 PMZachary Hughes
07/14/2020, 7:53 PMTask definition does not support launch_type FARGATE
errors I'm seeing online have to do with not properly setting requiresCompatibilities
. Can you confirm that you haven't set your agent's launch_type
to EC2 and try specifying FARGATE for requiresCompatibilities
in your task definition?Darragh
07/14/2020, 8:04 PMZachary Hughes
07/14/2020, 8:07 PMlaunch_type
, it should default to setting the appropriate requiresCompatibilities
field there. You may need to specify it for your flow environment, though.Darragh
07/14/2020, 8:10 PMZachary Hughes
07/14/2020, 8:25 PMrequiresCompatibilities
set?Darragh
07/14/2020, 9:46 PMrequiresCompatibilities
set on the envZachary Hughes
07/14/2020, 10:19 PMDarragh
07/14/2020, 10:54 PMZachary Hughes
07/15/2020, 4:34 PMDarragh
07/15/2020, 4:39 PMboto3.register_task_definition
was returning this, which shows it using the containerDefintions data from the agent, rather than the flowrun_task
Zachary Hughes
07/15/2020, 5:14 PMcontainerDefinitions
from the agent, you don't get your debug statement. Does the attempted run still fail with the same error we were discussing yesterday?Darragh
07/15/2020, 5:40 PMTask definition does not support launch_type FARGATE.')
I’ve removed the launch type from both the agent and the env, no luckZachary Hughes
07/15/2020, 6:58 PMTask definition does not support launch_type FARGATE
error you get when the requiresCompatibilities
field is clearly showing FARGATE.Darragh
07/15/2020, 7:22 PMFargateAgent
and LocalEnvironment(executor=LocalExecutor())
Zachary Hughes
07/15/2020, 7:37 PMGreg Desmarais
07/20/2020, 5:03 PMDarragh
07/20/2020, 5:58 PMZachary Hughes
07/20/2020, 8:55 PMrequiresCompatibilities
does seem to be properly set.Darragh
07/20/2020, 9:49 PMZachary Hughes
07/20/2020, 9:52 PMGreg Desmarais
07/21/2020, 4:11 AMDarragh
07/21/2020, 9:13 PMcontainerDefinitions
, and also has the property taskDefinition
set. taskDefinition
is set to my-flow
.
Now what I;ve just spotted is that this Flow [source above somewhere] actually creates 2 Task Definitions in ECS.
• Task Defintion A - my-flow
-> Correctly configured with 2 containers, etc…
• Task Definition B - parallell-collector
- this is the name of the overall Flow, but only contains 1 container, the one defined by the Docker Storage in the Flow
Seems a little odd that it would be creating 2 Task Definitions here - as an even more entertaining note, manually running my-flow
on Fargate actually succeeds, I can see the TaskDefinition run to completion and dig into it’s logs.
Any thoughts? I’m perfectly to accept an answer of “Oh your config is batshit, here’s what it should really look like” 🙂Zachary Hughes
07/22/2020, 3:31 PMflow.environment = FargateTaskEnvironment(...)
flow.environment.setup(flow)
flow.environment.execute(flow)
If you're running into issues with your Prefect environment, we can open an issue to take a look. But I'm not 100% sure what your flow should look like on the Fargate side.Darragh
07/22/2020, 8:48 PMTraceback (most recent call last):
File "flowRunner.py", line 44, in <module>
python_dependencies=args.python_dependencies, execution=args.execution)
File "./definitions/data/parallell/collector/parallellCollector.py", line 109, in main
flow.environment.execute(flow)
File "/Users/Darragh/.pyenv/versions/3.6.10/lib/python3.6/site-packages/prefect/environments/execution/fargate/fargate_task.py", line 309, in execute
**self.task_run_kwargs,
File "/Users/Darragh/.pyenv/versions/3.6.10/lib/python3.6/site-packages/botocore/client.py", line 316, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/Users/Darragh/.pyenv/versions/3.6.10/lib/python3.6/site-packages/botocore/client.py", line 635, in _make_api_call
raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidParameterException: An error occurred (InvalidParameterException) when calling the RunTask operation: Task definition does not support launch_type FARGATE.
Greg Desmarais
07/23/2020, 5:42 PMDarragh
07/23/2020, 5:47 PMZachary Hughes
07/23/2020, 5:58 PMTask definition does not support launch_type
issue, I think your best bet is to either open this issue up to the channel at large and see if anyone in the community has had it, or to see what AWS has to say. You're also welcome to open an issue so we can socialize and potentially figure out what's going on here!Greg Desmarais
07/23/2020, 6:03 PMdef get_aws_credentials() -> tuple:
"""
Retrieve the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY and return as a tuple
:return: tuple of AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
"""
home = os.path.expanduser("~")
creds_path = os.path.join(home, '.aws/credentials')
if not os.path.isfile(creds_path):
raise ValueError(f'You must have a configured ~/.aws/credentials file - see '
f'<https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html>')
config = configparser.ConfigParser()
try:
config.read(creds_path)
except Exception as ex:
logger.warning(f'Unable to read {creds_path}, are permissions correct?')
raise ex
return config['default'].get('aws_access_key_id'), config['default'].get('aws_secret_access_key')
def set_aws_credentials_env():
key_id, key = get_aws_credentials()
os.environ['AWS_ACCESS_KEY_ID'] = key_id
os.environ['AWS_SECRET_ACCESS_KEY'] = key
os.environ['REGION_NAME'] = DEFAULT_REGION
This also helped me avoid a token creation.TOKEN=`curl -s -X PUT "<http://169.254.169.254/latest/api/token>" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
EC2_IP=`curl -s -H "X-aws-ec2-metadata-token: $TOKEN" <http://169.254.169.254/latest/meta-data/local-ipv4>`
Darragh
07/23/2020, 6:49 PMGreg Desmarais
07/23/2020, 6:55 PMDarragh
07/23/2020, 6:57 PMlaunch_type FARGATE
issue I still don’t undertand why it’s creating 2 Task Definitions?
If it was at all possible to get a working example of creating 2 containers in the FargateTaskEnvironment that’d be great! If not I’ll just dump it to GitHub…launchType=FARGATE
but I don’t know exactly where yetdef __init__( # type: ignore
self,
launch_type: str = "FARGATE",
aws_access_key_id: str = None,
aws_secret_access_key: str = None,
aws_session_token: str = None,
region_name: str = None,
executor: "prefect.engine.executors.Executor" = None,
executor_kwargs: dict = None,
labels: List[str] = None,
on_start: Callable = None,
on_exit: Callable = None,
metadata: dict = None,
**kwargs,
) -> None:
self.launch_type = launch_type
# Not serialized, only stored on the object
self.aws_access_key_id = aws_access_key_id or os.getenv("AWS_ACCESS_KEY_ID")
self.aws_secret_access_key = aws_secret_access_key or os.getenv(
"AWS_SECRET_ACCESS_KEY"
)
self.aws_session_token = aws_session_token or os.getenv("AWS_SESSION_TOKEN")
self.region_name = region_name or os.getenv("REGION_NAME")
Zachary Hughes
07/23/2020, 7:20 PMDarragh
07/23/2020, 7:22 PMZachary Hughes
07/23/2020, 7:34 PM