https://prefect.io logo
Powered by
Title
b

Bharadwaj Yadati

06/20/2022, 5:41 AM
Hi , I’m planning to use prefect as a replacement of our internal framework, its worked like charm while using local agent but when i’m trying ecs agent , im facing issues , i want to run the same etl job on ec2 instances . The guide provided on prefect website (https://docs.prefect.io/orchestration/agents/ecs.html#running-ecs-agent-in-production) is not working with issue ( i even provided aws keys as env variables for task ), if i dont provide args it throws any error asking for it. I’m kind of stuck on this , kindly help me out. Thanks in advance
✅ 1
a

Anna Geller

06/20/2022, 10:09 AM
For ECS, check out this guide https://towardsdatascience.com/how-to-cut-your-aws-ecs-costs-with-fargate-spot-and-prefect-1a1ba5d2e2df
b

Bharadwaj Yadati

06/20/2022, 11:16 AM
i followed the same but got the following issue :
if i specify s3 path location i got the first issue. i’m kind of stuck , kindly help me out. @Anna Geller
a

Anna Geller

06/20/2022, 11:47 AM
It looks like you didn't follow this tutorial because then the agent wouldn't try to infer the default network configuration this way
can you start again from scratch, perhaps in Python 3.8 or 3.9?
network configuration is passed in this step explicitly: 
aws ecs create-service \
    --service-name $ECS_SERVICE_NAME\
    --task-definition $ECS_SERVICE_NAME:1 \
    --desired-count 1 \
    --launch-type FARGATE \
    --platform-version LATEST \
    --cluster $ECS_CLUSTER_NAME \
    --network-configuration awsvpcConfiguration="{subnets=[$SUBNET1, $SUBNET2, $SUBNET3],assignPublicIp=ENABLED}" --region $AWS_REGION
b

Bharadwaj Yadati

06/20/2022, 12:40 PM
yes , i created a cluster , created a service as above and registered a task , when task got registered , i got the above issue.
a

Anna Geller

06/20/2022, 12:54 PM
the best advice I have is to: • start from scratch and slowly follow the tutorial step-by-step and document all you do • document all the steps you have taken in some document, e.g. a markdown file • share all the steps you performed here (redact any sensitive info) e.g. as a GitHub Gist - this way it may be easier to find out at which point sth went wrong Have you tried again using a higher Python version as I suggested? Did you follow all the steps? At which point exactly did you encounter the error?
b

Bharadwaj Yadati

06/22/2022, 2:55 AM
i did all the mentioned steps but in vain , kindly check this https://gist.github.com/bharadwajaviso/6a0534d8710a586444639bb47306a2b4 after the task is created and service got created , task cannot find the service network configuration
Ideally this is not default vpc , this is our own vpc , i think there is a ticket opened for the same ? https://github.com/PrefectHQ/prefect/issues/5474
b

Blake Hamm

06/30/2022, 7:02 PM
@Bharadwaj Yadati were you able to troubleshoot this? I'm running into the same issue. Also, I am closely following the tutorial you have suggested @Anna Geller (except I'm trying to translate into terraform).
Also, from what I can tell here: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/ecs/agent.py#L250, this information is inferred from boto3 and ignoring the passed values from the AWS CLI/terraform/etc that's creating the service. I'm guessing that I need to setup secrets in prefect Cloud in order for it to pull environmental variables and infer my AWS credentials. I suspect the 
ec2 = get_boto_client("ec2", **self.boto_kwargs)
is returning an empty variable since I haven't configured my AWS credentials within the prefect ECS agent.
a

Anna Geller

07/01/2022, 12:29 PM
all credentials should be configured via IAM roles - key-value pairs won't work well with ECS and this is also security best practice to use IAM roles for ECS terraform, check out https://github.com/PrefectHQ/prefect-recipes/tree/main/aws/prefect/prefect-agent-on-ecs
👀 1
@Bharadwaj Yadati were you able to solve your networkConfiguration error?
b

Bharadwaj Yadati

07/01/2022, 1:16 PM
hi @Anna Geller , thanks for asking , not yet its still the same , just try running prefect ecs agent from local even it requires networkconfiguration if we have non-default vpc fargate cluster . locally i’m providing the network configuration file from local path , but with agent on fargate it fails.
a

Anna Geller

07/04/2022, 1:14 PM
do you know how to proceed? are you stuck? I'd encourage you to take it all step by step and look at which component doesn't work as expected, you can redeploy everything from scratch using that terraform template
b

Bharadwaj Yadati

07/05/2022, 5:06 AM
we are planning to switch to Sagemaker as its easy to set up and managed service and started porting our code.
a

Anna Geller

07/05/2022, 11:10 AM
you mean switching from Prefect to Sagemaker? LMK if there's anything else we can help with I'm currently working on automated ECS recipe that I will contribute to this repo once I'm finished https://github.com/anna-geller/prefect-dataops-ecs
b

Blake Hamm

07/05/2022, 5:59 PM
@Anna Geller thank you for sending that recipe! 🙏 It worked fantastic. Are there any resources on opinions on docker agent in autoscaling ec2 vs ecs agent? It sounds like the docker agent is more commonly used by the community?
a

Anna Geller

07/05/2022, 6:05 PM
for Prefect 2.0 I will provide more guidance on that very soon The main difference here is that with EC2 you need to scale resources yourself and with ECS Farage it's serverless and AWS takes care of scaling resources for you
✅ 1
b

Blake Hamm

07/05/2022, 6:06 PM
Thank you Anna. I really appreciate your support!
b

Bharadwaj Yadati

07/06/2022, 11:18 AM
thank you @Anna Geller , much appreciated , this will help us to accelerate our work , we have an in house orchestration service running for past 6 years , we thought of retiring it and switching to Prefect so evaluating if .As we run most of our jobs on ec2 or dockers on ecs , I tried with Ray , Prefect and Airflow but picked Prefect , as the code changes are less and migration is smoother . if either EC2 support or Fargate support is provided , love to use it. thanks once again.
🙌 1
4 Views
#prefect-communityJoin Slack
Powered by