Kesav Kolla

06/01/2020, 7:09 PM
I see the perfect tasks are not long running. Eg: the database fetch task it opens connection executes query and closes connection. Is it possible to develop a task which runs forever. I can write a PostgreSQL Task which initializes connection pool and it's open forever. All subsequent query executions will re-use connection from pool. Another use case is a gRPC or Arrow Flight IPC server which runs forever and will serve other tasks the necessary data. Something similar to zeebe or netflix conductor provides.

Chris White

06/01/2020, 7:24 PM
Because Prefect is built on the model of workflows-as-DAGs, we require that tasks finish before downstream dependencies can begin. That being said, we are exploring a notion of “Resource” that you can specify on your Flow that will essentially achieve what you’re after (essentially we’d be introducing a new non-task abstraction). This feature is not currently prioritized so probably won’t be picked up until next quarter