Hi, I was wondering if anyone has any great ideas for how to deal with cancellation of long running...
j
Hi, I was wondering if anyone has any great ideas for how to deal with cancellation of long running queries started from a Prefect task. In my case I’m dealing with PostgreSQL and while psycopg2 will try to gently stop a query when you close a connection, in many cases the reason we are cancelling a flow/task is because that query has been running for too long and is unlikely to finish naturally in an acceptable timeframe. Instead of having to manually kill the query with
pg_terminate_backend
it would be nice if the task getting cancelled would be able to call
pg_terminate_backend
.
k
Maybe you can use a flow level state handler to call that if the flow is being cancelled?
Actually sorry I believe I tried this before and it didn't work unfortunately. The Flow process just stops.
j
Yeah, it’s tricky on both the Prefect end and PostgreSQL end, cause you need to store the process ID somewhere just in case you need to kill it.