What’s the minimum delay I could expect between tr...
# ask-communitye
Edmund Tian
12/14/2022, 4:49 AMWhat’s the minimum delay I could expect between triggering a Flow run and the first Task starting execution? And what’s the minimum delay I could expect between a Task finishing execution and the next Task in the Flow starting execution?
Context:
Every time a new user of my application is created, I want to trigger a Flow run that will scrape this user’s data from multiple sources and populate our database. I need to populate my database within 30 seconds of user creation. This is achievable with my current implementation of orchestrating this pipeline via a python function.
I want to make sure that migrating my orchestration logic to Prefect will not add significant time to this process. Ideally I’d keep the additional time under 5 seconds. Is this achievable with Prefect?
p
Peyton Runyan
12/14/2022, 2:06 PMThis is going to heavily depend on the infrastructure you're running on
Peyton Runyan
12/14/2022, 2:07 PMis your current function just being executed on the same server your API is running on? If so, you could literally just decorate your existing function function with
@flowPeyton Runyan
12/14/2022, 2:11 PM Copy code
from prefect import flow
@flow
def cat():
print("meow!")
if __name__ == "__main__":
cat()Peyton Runyan
12/14/2022, 2:19 PMI just tested it with prefect cloud, which is probably what you'll wind up using in production. It adds about 3 seconds compared to running a raw function on the same machine. If you're making a network call to a worker to kick off the functions that you're using currently, then the difference will be smaller.
Peyton Runyan
12/14/2022, 2:23 PMIf you assume 2 to 3 seconds of overhead per non-concurrent task, you probably have a decent model of the cost of using cloud. Given that all of your work is scraping sites, that's all concurrent tasks.
If you do a flow with a few concurrent scrapers, you could get pretty full-fledged orchestration for something like 6 to 7 seconds of additional overhead.
Otherwise you can flow-level orchestration for the 3 seconds of overhead using just the flow decorator on your existing code.
e
Edmund Tian
12/16/2022, 5:51 AMThanks for the comprehensive answer! I didn’t realize it was as easy as just adding the @flow decorator. So what’s the advantage of using Prefect Cloud? Just the UI?
Edmund Tian
12/16/2022, 6:45 AMNvm, from reading more of the docs, looks like just decorating my python functions is all I need. Thanks!
👍 1