Hi just moving this over from the wrong channel: I recently finished porting one of our pipelines over to prefect and have been eager to move out of local and over to an EKS/ECR configuration. I saw some guides by Anna and Jimmy that were really helpful so far, but I've hit multiple collisions and was seeking some help to work through some of them. Specifically two blockages I've encountered involve using our own custom base image, which is hosted on ECR, which is actually pulling code from CodeArtifact; that approach is hitting healthcheck issues when building. Another issue I'm having is being unable to create new EKS clusters through eksctl (where CloudFormation spits out max # of elastic IPs reached) , but I'm assuming that's more of a provisioning/permissioning issue and for that I'm just wondering if anyone else has gotten stuck at these points as well. I can however deploy an EKS Agent and view it in kubectl get pods, and publish a flow on Prefect Cloud regardless of these blockers, so I feel like I'm tangled up in the last few steps in getting this to work. (Thanks Dylan for pointing me to this channel!)

Hello Charles! 👋 Can you share more information about healthcheck issue you are facing (any errors, logs, etc). Also, there is a limit set by AWS for Elastic IPs (5 IPs per account), but you can submit a request for Elastic IPs to Amazon.

So looking closer at the healthcheck it's actually erroring out when attempting to run "aws codeartifact login"; perhaps I'm implementing awscli incorrectly?

Hi Charles!

aws codeartifact login

probably complains because AWS credentials are not configured properly (sorry, I'm not codeartifact expert). And yes, you're right, basic template looks like this:

from prefect.storage import Docker
from prefect.run_configs import KubernetesRun
from prefect import task, Flow
import prefect

STORAGE = Docker(
    registry_url="<>.dkr.ecr.<>.<http://amazonaws.com|amazonaws.com>",
    image_name="<>",
    python_dependencies=["<>"],
)

RUN_CONFIG = KubernetesRun()

@task
def print_me():
    pass

with Flow(
    "ecs-with-ecr-example", storage=STORAGE, run_config=RUN_CONFIG
) as flow:
    print_me()

flow.register(project_name="<>")

So when I try to pass ECSRun(), it says run_config only supports KubernetesRun(); any idea what that means? As in, the flow will not deploy when I use ECSRun as the run_config while running a Kubernetes Agent.

I apologize, misread ECS as EKS. You need to use KubernetesRun instead ECSRun 🙂

Alright i'm going to give it a whirl and report back thank you!

Sounds good!

So I (think) i got one successful dummy run (saw a resource spike on EC2 and also my print statement in Prefect logs) but no more after that