https://prefect.io logo
Title
m

Michael Ludwig

09/05/2020, 10:20 PM
Hi all, I am would be curious if you at one point maybe thought about providing a completly serverless Prefect agent option? We currently had one tiny service in ECS Fargate with one container running the Fargate agent around the clock. As I am very much a serverless fan and I might only have a daily scheduled flow I thought it would be nice to only activate the agent when needed or even have a push-based system. On AWS side I would see two options e.g. a Lambda function which is triggered via a webhook over AWS Api Gateway or even a AWS SNS webhook. Alternatively if push is not possible I could imagine a poll-based cron scheduled lambda to check for flow runs. Maximum frequency is unfortunately only one minute but might be ok for certain cases but that would be anyways not really “on-demand”. To my knowledge the agent is anyways doing the flow runs in fire and forget style therefore I thought this could be quite doable. Anybody with more insights to the agent having an opinion or did think about something similiar already? 🙂 (I hope it is not wrong to post this question in this channel 😄 )
j

Jeremiah

09/07/2020, 1:55 PM
Hi Michael - it’s on the roadmap but requires some new architecture. You’ll see the very first steps in this direction start to emerge in the next backend release though, with improved agent management.
My personal goal is to do serverless in a “push” based way :)
m

Michael Ludwig

09/07/2020, 2:45 PM
Wow @Jeremiah, really excited for this. Sounds great 🙂
j

Jeremiah

09/07/2020, 2:45 PM
Yup! Just to be clear it might be a short while until it’s 100% in place but it’s definitely roadmapped