I appreciate this might be a fairly significant engineering effort, so not something I'm holding my breath for, but is there any appetite for Lambda-based task execution? We're probably heading for ECS tasks as a serverless execution environment, but they're kind of slow to spin up, and for short-lived tasks it feels like Lambda would be at the right sort of level

Lambda is very limited in what you can run on this, max 15 min runtime. It's free and great for event-driven execution of small actions but it's not great as a runtime environment. If you would like to take a stab at contributing an infra block to prefect-aws repo, we would most likely accept it and help you get it released using prefect with serverless framework for lambda works pretty seamless without infra blocks https://medium.com/the-prefect-blog/event-driven-data-pipelines-with-aws-lambda-prefect-and-github-actions-b3d9f84b1309

Yes, I appreciate it wouldn't be a good general-purpose runner, but many of our tasks are actually quite short-lived (especially with the prefect 2 pricing model encouraging smaller tasks), and it would be nice to be able to use prefect for orchestrating and monitoring these as well as our bigger batch tasks

You may be able to use lambda with subprocess infrastructure and an agent with the

--run-once

flag

Sorry it's possible I'm not using the right terminology -- I imagine that the agent would probably be on long-lived infrastructure. And then each flow run would be on a new lambda invocation

👍 yep

Cool, well I'm not going to make any commitments but if it starts to look like that would make sense for us then I'll reach back out

Sweet! We’ll probably do it eventually too.

@Christopher, have you thought about setting up ECS on EC2 for lower latency task runs?

No, in fact my ECS experience is limited to Fargate. Is that something I should look into?

well, its an option to consider. I havent dug too deep into it either

Please report back if you find it transformative!

yes, that would be a simpler way to start out

who knows, maybe AWS will surprise us and will announce some cool caching on Fargate this week during re:Invent 😄 would be sweet

That would be amazing. We would definitely pay for that feature

true, if they can do it for lambda, they might do it for fargat too!

there is one trick to slightly reduce the latency of ECSTask block and it is to use a custom task definition ARN - this way Prefect doesn't have to register a new task definition and deregister it, so there is slightly less latency. From my QA it was max 30s which seems acceptable for batch processes and even near-real time

@Christopher have you looked into this further? I found this thread because my team is in the same place and I'm thinking about trying to contribute a lambda infra block. If you have any ideas or hit any roadblocks I'd be happy to hear them

We didn't I'm afraid, got distracted by other fires to put out! I still think it's interesting though