Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi - I have a flow that runs under a minute and is part of another flow.  I was thinking of wrapping the flow in an aws lambda to speed up the overall processing of the larger flow.  Any thoughts?

Hey <@UNSQ0T2V9>, how long is the larger flow? You could but I’m not sure it’s worth the lambda dependency unless you have other lambda components already.

Small - I am trying to work around large cold start times. Actually the thought is to use lambda for the second flow also. Basically I want to use prefect to orchestrate some lamdas so I don’t need to spin up large Dask clusters.  The first flow is mapped over, the second flow is reduce.

Wouldn't lambda also have a cold start depending how often you run the flow?

It would but because it is mapped over *and* iterated over 10 -20 times I think it might be faster by treating it as an "app"

<@U01QEJ9PP53> If I were to wrap a prefect flow in a lambda - What agent would I use? Local.  I have a small ec2 instance in the cloud where I host the prefect server.