https://prefect.io logo
Join Slack
Powered by
Hi , I’m planning to use prefect as a replacement ...
# prefect-community
b
Hi , I’m planning to use prefect as a replacement of our internal framework, its worked like charm while using local agent but when i’m trying ecs agent , im facing issues , i want to run the same etl job on ec2 instances . The guide provided on prefect website (https://docs.prefect.io/orchestration/agents/ecs.html#running-ecs-agent-in-production) is not working with issue ( i even provided aws keys as env variables for task ), if i dont provide args it throws any error asking for it. I’m kind of stuck on this , kindly help me out. Thanks in advance
✅ 1
a
b
i followed the same but got the following issue :
if i specify s3 path location i got the first issue. i’m kind of stuck , kindly help me out. @Anna Geller
a
It looks like you didn't follow this tutorial because then the agent wouldn't try to infer the default network configuration this way
can you start again from scratch, perhaps in Python 3.8 or 3.9?
network configuration is passed in this step explicitly:
Copy code
aws ecs create-service \
    --service-name $ECS_SERVICE_NAME\
    --task-definition $ECS_SERVICE_NAME:1 \
    --desired-count 1 \
    --launch-type FARGATE \
    --platform-version LATEST \
    --cluster $ECS_CLUSTER_NAME \
    --network-configuration awsvpcConfiguration="{subnets=[$SUBNET1, $SUBNET2, $SUBNET3],assignPublicIp=ENABLED}" --region $AWS_REGION
b
yes , i created a cluster , created a service as above and registered a task , when task got registered , i got the above issue.
a
the best advice I have is to: • start from scratch and slowly follow the tutorial step-by-step and document all you do • document all the steps you have taken in some document, e.g. a markdown file • share all the steps you performed here (redact any sensitive info) e.g. as a GitHub Gist - this way it may be easier to find out at which point sth went wrong Have you tried again using a higher Python version as I suggested? Did you follow all the steps? At which point exactly did you encounter the error?
b
i did all the mentioned steps but in vain , kindly check this https://gist.github.com/bharadwajaviso/6a0534d8710a586444639bb47306a2b4 after the task is created and service got created , task cannot find the service network configuration
Ideally this is not default vpc , this is our own vpc , i think there is a ticket opened for the same ? https://github.com/PrefectHQ/prefect/issues/5474
b
@Bharadwaj Yadati were you able to troubleshoot this? I'm running into the same issue. Also, I am closely following the tutorial you have suggested @Anna Geller (except I'm trying to translate into terraform).
Also, from what I can tell here: https://github.com/PrefectHQ/prefect/blob/master/src/prefect/agent/ecs/agent.py#L250, this information is inferred from boto3 and ignoring the passed values from the AWS CLI/terraform/etc that's creating the service. I'm guessing that I need to setup secrets in prefect Cloud in order for it to pull environmental variables and infer my AWS credentials. I suspect the 
ec2 = get_boto_client("ec2", **self.boto_kwargs)
is returning an empty variable since I haven't configured my AWS credentials within the prefect ECS agent.
a
all credentials should be configured via IAM roles - key-value pairs won't work well with ECS and this is also security best practice to use IAM roles for ECS terraform, check out https://github.com/PrefectHQ/prefect-recipes/tree/main/aws/prefect/prefect-agent-on-ecs
👀 1
@Bharadwaj Yadati were you able to solve your networkConfiguration error?
b
hi @Anna Geller , thanks for asking , not yet its still the same , just try running prefect ecs agent from local even it requires networkconfiguration if we have non-default vpc fargate cluster . locally i’m providing the network configuration file from local path , but with agent on fargate it fails.
a
do you know how to proceed? are you stuck? I'd encourage you to take it all step by step and look at which component doesn't work as expected, you can redeploy everything from scratch using that terraform template
b
we are planning to switch to Sagemaker as its easy to set up and managed service and started porting our code.
a
you mean switching from Prefect to Sagemaker? LMK if there's anything else we can help with I'm currently working on automated ECS recipe that I will contribute to this repo once I'm finished https://github.com/anna-geller/prefect-dataops-ecs
b
@Anna Geller thank you for sending that recipe! 🙏 It worked fantastic. Are there any resources on opinions on docker agent in autoscaling ec2 vs ecs agent? It sounds like the docker agent is more commonly used by the community?
a
for Prefect 2.0 I will provide more guidance on that very soon The main difference here is that with EC2 you need to scale resources yourself and with ECS Farage it's serverless and AWS takes care of scaling resources for you
✅ 1
b
Thank you Anna. I really appreciate your support!
b
thank you @Anna Geller , much appreciated , this will help us to accelerate our work , we have an in house orchestration service running for past 6 years , we thought of retiring it and switching to Prefect so evaluating if .As we run most of our jobs on ec2 or dockers on ecs , I tried with Ray , Prefect and Airflow but picked Prefect , as the code changes are less and migration is smoother . if either EC2 support or Fargate support is provided , love to use it. thanks once again.
🙌 1
4 Views
Open in Slack
PreviousNext
Powered by