Channels
#prefect-community
Title
# prefect-community
e
Edmund Tian
12/14/2022, 4:49 AMWhat’s the minimum delay I could expect between triggering a Flow run and the first Task starting execution? And what’s the minimum delay I could expect between a Task finishing execution and the next Task in the Flow starting execution?
Context:
Every time a new user of my application is created, I want to trigger a Flow run that will scrape this user’s data from multiple sources and populate our database. I need to populate my database within 30 seconds of user creation. This is achievable with my current implementation of orchestrating this pipeline via a python function.
I want to make sure that migrating my orchestration logic to Prefect will not add significant time to this process. Ideally I’d keep the additional time under 5 seconds. Is this achievable with Prefect?
p
Peyton Runyan
12/14/2022, 2:06 PMThis is going to heavily depend on the infrastructure you're running on
is your current function just being executed on the same server your API is running on? If so, you could literally just decorate your existing function function with
@flow Copy code
from prefect import flow
@flow
def cat():
print("meow!")
if __name__ == "__main__":
cat()I just tested it with prefect cloud, which is probably what you'll wind up using in production. It adds about 3 seconds compared to running a raw function on the same machine. If you're making a network call to a worker to kick off the functions that you're using currently, then the difference will be smaller.
If you assume 2 to 3 seconds of overhead per non-concurrent task, you probably have a decent model of the cost of using cloud. Given that all of your work is scraping sites, that's all concurrent tasks.
If you do a flow with a few concurrent scrapers, you could get pretty full-fledged orchestration for something like 6 to 7 seconds of additional overhead.
Otherwise you can flow-level orchestration for the 3 seconds of overhead using just the flow decorator on your existing code.
e
Edmund Tian
12/16/2022, 5:51 AMThanks for the comprehensive answer! I didn’t realize it was as easy as just adding the @flow decorator. So what’s the advantage of using Prefect Cloud? Just the UI?
Nvm, from reading more of the docs, looks like just decorating my python functions is all I need. Thanks!
👍 1