Hello I have seen an interesting to me error my code attempt

Question

Hello I have seen an interesting to me error my code attempts to get the value of a secret basically like this ``` secret = prefect client secrets Secret SFTP PASSWORD SECRET NAME return secret get ``` and it fails with a `prefect exceptions ClientError` from the GraphQL API

Ben Ayers-Glassey · Answer

```Task download shapefile from sftp 1684 Exception encountered during task execution Traceback most recent call last File usr local lib python3 8 site packages prefect client secrets py line 140 in get value = secrets self name KeyError ZESTY REGRID SFTP PASSWORD During handling of the above exception another exception occurred Traceback most recent call last File usr local lib python3 8 site packages prefect engine task runner py line 880 in get task run state value = prefect utilities executors run task with timeout File usr local lib python3 8 site packages prefect utilities executors py line 468 in run task with timeout return task run args kwargs type ignore File home zesty bag repos zestyai data ingestion flows parcels regrid sftp parcels regrid sftp py line 288 in download shapefile from sftp File project regrid py line 49 in get sftp client password = get sftp password File project regrid py line 29 in get sftp password return secret get File usr local lib python3 8 site packages prefect client secrets py line 164 in get raise exc File usr local lib python3 8 site packages prefect client secrets py line 148 in get result = self client graphql File usr local lib python3 8 site packages prefect client client py line 464 in graphql raise ClientError result errors prefect exceptions ClientError path secret value message An unknown error occurred extensions code INTERNAL SERVER ERROR ```

Ben Ayers-Glassey · Answer

This is a mapped task with thousands of child tasks and it has been using a lot of memory so perhaps that s the reason somehow But that seems strange because these errors were happening long before the machine ran out of memory entirely FWIW I ve seen a variant where one task logged an almost identical stack trace except the error contained API ERROR instead of INTERNAL SERVER ERROR ```prefect exceptions ClientError path secret value message Unable to complete operation An internal API error occurred extensions code API ERROR ```

Ben Ayers-Glassey · Answer

Any ideas for why this might be happening One thought I had was API throttling like maybe my mapped task is generating so many child tasks 3 500 or so which are all attempting to grab the same secret so the GraphQL secret value endpoint is throttling me But I m using a LocalDaskExecutor limited to 5 workers and these tasks are taking at least a few seconds each so it s not like I m hammering the API many times every second or anything thinking face

Ben Ayers-Glassey · Answer

Ah in fact I see this has also happened with another flow which doesn t use much memory at all We re using VertexAgent so each flow run is spun off as a separate <http Vertex ai|Vertex ai> job So the memory intensive flow can t have been affecting the other s memory usage So I think it must be something like throttling or an unrelated temporary API issue

Ben Ayers-Glassey · Answer

Wow actually I see this same issue with getting a secret s value popping up last Saturday So this is just looking like an intermittent issue with getting secrets from Prefect Cloud

Bianca Hoch · Answer

Hey Ben You re spot on in your assessment We do have on ongoing investigation into the issues revolving around retrieving secrets Here is our <https prefect status io |status page> which we are updating in in real time during this process

Ben Ayers-Glassey · Answer

Ah phew so it s not just us Ok I ll check the status page Thank you