David Elliott
02/03/2023, 6:32 PMTask run 'xxxx' received abort during orchestration
which I note has a TODO against it RE discovering why it happens. Details in 🧵Task run '6f4c3941-1a44-4ed4-8c05-f5646c7ef8e7' received abort during orchestration: Error validating state: DBAPIError("(sqlalchemy.dialects.postgresql.asyncpg.Error) <class 'asyncpg.exceptions.QueryCanceledError'>: canceling statement due to statement timeout") Task run is in RUNNING state.
I hit 10 of these in my large flow run just now - I ran 3600 tasks in about 30mins, max concurrency of 100 tasks at a given time.Version: 2.7.11
API version: 0.8.4
Python version: 3.8.7
Git commit: 6b27b476
Built: Thu, Feb 2, 2023 7:22 PM
OS/Arch: darwin/x86_64
Profile: bi-p
Server type: cloud
Zanie
02/03/2023, 7:14 PMDavid Elliott
02/03/2023, 8:04 PMTask run '80dc471b-bd97-47d2-9f54-fb8707551b39' received abort during orchestration: Error validating state: DBAPIError("(sqlalchemy.dialects.postgresql.asyncpg.Error) <class 'asyncpg.exceptions.QueryCanceledError'>: canceling statement due to statement timeout") Task run is in RUNNING state.
It’s mid flow-run (flow is still running) and was thrown at task run end time. Wonder if it’d be worth adding a retry for a PG timeout error? Also, is the PG timeout common / expected..?Zanie
02/06/2023, 6:31 PMPREFECT_ORION_
prefixed settings only affect the server (i.e. setting it client-side has no effect)David Elliott
02/06/2023, 6:33 PMZanie
02/06/2023, 6:44 PMDavid Elliott
02/06/2023, 6:45 PMZanie
02/07/2023, 3:50 PMAleksandr Liadov
02/08/2023, 10:28 AMZanie
02/08/2023, 3:51 PMAleksandr Liadov
02/09/2023, 4:18 PMTask run 'c34bdce0-743d-45ec-a387-33e7f17464be' received abort during orchestration: Error validating state: DBAPIError("(sqlalchemy.dialects.postgresql.asyncpg.Error) <class 'asyncpg.exceptions.QueryCanceledError'>: canceling statement due to statement timeout") Task run is in RUNNING state.
Zanie
02/09/2023, 4:30 PMDavid Elliott
02/13/2023, 9:40 AMZanie
02/13/2023, 3:32 PMDavid Elliott
02/13/2023, 6:36 PMStéphan Taljaard
02/13/2023, 6:43 PMZanie
02/13/2023, 7:21 PMDavid Elliott
02/13/2023, 7:55 PMKalise Richmond
02/23/2023, 5:34 PM