What s the minimum delay I could expect between triggering a

Question

What s the minimum delay I could expect between triggering a Flow run and the first Task starting execution And what s the minimum delay I could expect between a Task finishing execution and the next Task in the Flow starting execution Context Every time a new user of my application is created I want to trigger a Flow run that will scrape this user s data from multiple sources and populate our database I need to populate my database within 30 seconds of user creation This is achievable with my current implementation of orchestrating this pipeline via a python function I want to make sure that migrating my orchestration logic to Prefect will not add significant time to this process Ideally I d keep the additional time under 5 seconds Is this achievable with Prefect

Peyton Runyan · Answer

This is going to heavily depend on the infrastructure you re running on

Peyton Runyan · Answer

is your current function just being executed on the same server your API is running on If so you could literally just decorate your existing function function with ` flow` and it should be well within your needs

Peyton Runyan · Answer

```from prefect import flow flow def cat print meow if name == main cat ```

Peyton Runyan · Answer

I just tested it with prefect cloud which is probably what you ll wind up using in production It adds about 3 seconds compared to running a raw function on the same machine If you re making a network call to a worker to kick off the functions that you re using currently then the difference will be smaller

Peyton Runyan · Answer

If you assume 2 to 3 seconds of overhead per non concurrent task you probably have a decent model of the cost of using cloud Given that all of your work is scraping sites that s all concurrent tasks If you do a flow with a few concurrent scrapers you could get pretty full fledged orchestration for something like 6 to 7 seconds of additional overhead Otherwise you can flow level orchestration for the 3 seconds of overhead using just the flow decorator on your existing code

Edmund Tian · Answer

Thanks for the comprehensive answer I didn t realize it was as easy as just adding the flow decorator So what s the advantage of using Prefect Cloud Just the UI

Edmund Tian · Answer

Nvm from reading more of the docs looks like just decorating my python functions is all I need Thanks