https://prefect.io logo
Join Slack
Powered by
Hi Prefect devs, I've been working on a POC and g...
# ask-community
b
Hi Prefect devs, I've been working on a POC and got everything running with ECS and fargate. I have what might be a fairly unusual use case where I am scheduling flow runs (through the grqphql api) for a day in specific intervals to scrape certain data. These intervals are spread out and then as the event nears become closer (with flows running every 10 seconds as the event gets closer) (think of horse racing data as the race gets closer ). Anyway, with ECS scheduling close running jobs doesn't really work out as the start up time is for the task definition is ~ 45 seconds. A possible solution for this would be to give us the ability to run these jobs on Lambdas (they now support Docker image runs and we could keep the lambda warm for shorter start up times). Wondering if this is something that Prefect might consider in the future or is this just too much of a corner case and I'm missing the point on what Prefect should be used for?
k
Hey @Ben Muller, I think there might be some components that don’t fit well together, but work around-able. First, though the ability to run on Lambdas might be something you can open a feature request for in the repo. The following are just a couple of thoughts but I don’t have hard suggestions.
First, you can use the AWS Tasks in the task library to trigger a lambda. You may lose some Prefect features though (results, secrets, traceability) but it might work well for your use case. You can use the Lambda payload to provide input from the Prefect Flow.
Prefect at sub-minute flows is not optimal because there is some startup time required. In that sense, you can rearrange your logic to run more times within the same Flow. Prefect has a 
LOOP
signal that can be used to loop a task, You can also do this with Python recursion inside a task. This in general would prevent the repeated spin ups and downs
b
The flow is executed by ECS though, so everytime the flow is executed it takes 45 seconds for the task definition and the image to be pulled. That's where the issue lies...
Ah ok, your second suggestion might work well.
Thinking prefect might not be the correct platform as I'm needing exact spin up times. Love the platforms observability though.. So was hoping to use it for more than just an Airflow alternative
k
Oh you’re right. Maybe then you’d have to move off of the ECS agent and use a Local agent on an EC2 to invoke the Lambdas.
This would decrease the spin up and down right?
Where is the ECS agent running though? Is there something stopping you from replacing that with a Local agent that hits Lambdas? (I guess it’s the traceability of the setup)
s
Fargate is the issue 😅 If you were to have an existing ECS cluster where the container was already downloaded, it would be much snappier.
upvote 1
🙌 2
k
Thanks for chiming in @Spencer!
6 Views
Open in Slack
PreviousNext
Powered by