https://prefect.io logo
Title
r

Rikimaru Yamaguchi

02/27/2023, 8:11 AM
Hello! Is there a way to run ECSTask on a SPOT instance? thank you
r

Rob Freedy

02/27/2023, 2:53 PM
You can use
launch_type
in ECSTask to specify
FARGATE_SPOT
(see docs here)
c

Christopher Boyd

02/27/2023, 8:08 PM
While this is certainly doable, there are some open issues regarding scheduling of work if the spot instances are reclaimed while your work is still processing
r

Rikimaru Yamaguchi

02/28/2023, 1:02 AM
thank you It doesn’t work, do you know why?
c

Christopher Boyd

02/28/2023, 1:03 AM
r

Rikimaru Yamaguchi

02/28/2023, 1:04 AM
ohhh
sorry
c

Christopher Boyd

02/28/2023, 1:04 AM
all good, if that’s all that’s needed, that’s the easiest answer I can give all day 🙂
r

Rikimaru Yamaguchi

02/28/2023, 1:05 AM
it’s work! thank you!
🙌 1
one more question I have set env in ECSTask. However, various credential information is visible to people who can log in to the perfect cloud. Are there any other good methods? I considered using the aws parameter store, but gave up because AWS_SECRET_KEY must be placed in ECSTask blocks env.
c

Christopher Boyd

02/28/2023, 1:11 AM
I’ll have to think and get back to you on this one in the morning. Generally RBAC and roles are the way to limit who can see what - if a user has access to the flow and history, they can see what goes in. You can use Prefect secrets which are obscured, but that won’t obscure what ECS reports back
I’ll check with the team if there is a practice for doing so
r

Rikimaru Yamaguchi

02/28/2023, 1:13 AM
thank you. looking forward to it.
c

Christopher Boyd

02/28/2023, 1:36 PM
The safest route is to pass secrets from AWS secrets manager into the task definition as env vars. This is what we do with the prefect API key in our recipe: https://github.com/PrefectHQ/prefect-recipes/blob/main/devops/infrastructure-as-code/aws/tf-prefect2-ecs-agent/ecs.tf#L36-L41 (credit to @Emil Christensen)
j

Jing Xie

04/20/2023, 4:06 PM
Hi, I just introduced myself to this community and came across this thread re: ECS and using spot instances with Prefect. Memory Machine automatically manages spot instance reclaims and also has an user defined interval based checkpoint and recovery service that works with any OCI compliant container. This enables any job running on spot to be resumed on a new compute instance vs starting over, and it is fully automated from job submit to job complete. If this sounds like a useful approach I’d love to learn more @Rikimaru Yamaguchi .