Jaime Raldua Veuthey

04/26/2023, 8:55 AM
Hi, I have a pipeline in AWS where things trigger each other and would like to move it entirely to prefect but have some doubts/questions: 1. There is a Lambda_1 triggered on a schedule via cron job in Eventbridge. At the end this Lambda_1 has some python code to trigger Lambda_2 a. Q1.1) Can I trigger Lambda_1 in a Prefect deployment? b. Q1.2) Can Lamba_2 be triggered as well from Prefect depending on the completion status of Lambda_1? 2. There is a Lambda_2 that at the end of its python code does the following: updates the Fargate service for Container_1 to desired count = 1 (basically starts a task that wil be billed starting then). a. Q2.1) Can I update, with Prefect, the service for Container_1 to have 1 task running, depending on the completion status of Lambda_2? b. Q2.2) Can I start, with Prefect, the Prefect deployment for Container_1 depending on the completion status of the service update? c. Q2.3) Can I, with Prefect, update the service again to 0 so that it stops the billing? this should also depend on the completion status. If all python code was succesfull then stop and if not then retry a couple of times the flow for the deployment in Container_1 3. The Container_1 has a Prefect deployment that at the end calls Lambda_3 and Lambda_4 4. Lambda_3 is similar to Lambda_4 and it starts updates another fargate service to have 1 task running. Then it should start the Prefect deployment inside Container_2 5. Container_2 is similar to Container_1. It should call another lambda if the python code run succesfully and should retry a couple of times if there was any errors. At the moment I have all these triggers/calls within the codes and have to no proper way to see unified logs for everything so I thought Prefect could help and simplify my experience. General suggestions on the overall architecture are also welcome :)Thanks!

Mike Grabbe

04/26/2023, 11:52 AM
In short, anything you can do in boto3, you can orchestrate with prefect.
You could transition most of this orchestration to prefect: start execution at on a set schedule, invoke a lambda, trigger an ecs task, poll the status of each, and then act on the response once you receive a fail/success result.