Channels
#prefect-cloud
Title
e
Edmondo Porcu
06/28/2023, 12:24 AMHello world. Does the cloud edition capture identity information with the flow runs?
n
nicholas
06/28/2023, 12:30 AM
Hi @Edmondo Porcu - do you mean information about who created runs?
e
Edmondo Porcu
06/28/2023, 12:43 AMyes
given the open source version has no authentication
n
nicholas
06/28/2023, 1:16 AM
Got it - yes, Cloud tracks captures that information in a few ways; the actor (user or service account) responsible for creating a given flow run is referenced in flow run details (this is visible in the app). For organizations, this information is also captured in the form of audit logs.
e
Edmondo Porcu
06/28/2023, 2:24 AMby the way, do you know that the recent docker images do not work if you do
docker run prefect:whatever prefect server startn
nicholas
06/28/2023, 2:25 AM
That doesnât sound good - please open a ticket in the prefect repo with some info about what youâre trying, including some system info if youâre able so we can triage that appropriately
e
Edmondo Porcu
06/28/2023, 2:28 AMThanks. Maybe you do have a better workaround for our problems. We are evaluating prefect using hte Helm Chart before purchasing the cloud version (we do have a quote already)
We have to run a production job, but we havenât got prefect deployed in production kubernetes yet, so we want to run it âserverlessâ. However, if we run it serverless, it seems kind of impossible to submit in parallel ten flows to databricks:
⢠There is a locking error on the SQLLite
⢠they do not proceed in parallel
As a workaround I am trying to start a local prefect server within a Github Workflows
btw, I finally managed to start prefect on Github Workflows, but now I get the following error:
Copy code
02:33:17.023 | INFO | Flow run 'attentive-wildcat' - View at <http://localhost:4200/flow-runs/flow-run/f01d5f08-1302-440b-8263-10a93bdbc572>
02:33:17.218 | ERROR | Flow run 'vivid-donkey' - Crash detected! Execution was interrupted by an unexpected exception: PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<http://localhost:4200/api/block_documents/>'
Response: {'exception_message': 'Internal Server Error'}
For more information check: <https://httpstatuses.com/500>
02:33:17.244 | ERROR | Flow run 'big-vole' - Crash detected! Execution was interrupted by an unexpected exception: PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<http://localhost:4200/api/block_documents/>'
Response: {'exception_message': 'Internal Server Error'}
For more information check: <https://httpstatuses.com/500>n
nicholas
06/28/2023, 2:36 AM
It sounds like you might be hitting the limit of what a database like SQLite can offer - as mentioned in the docs, production use cases (or just ones with heavy parallelism) will usually require Postgres. The helm chart contains a subchart to help with this
e
Edmondo Porcu
06/28/2023, 2:36 AMYeah I am now starting the local deployment with
prefect server startHowever I am not using blocks, and I have no idea why there is that error
It sounds extraordinarily complicated given that we need to kick 10 databricks jobs in parallel
Do you think underlying the 500 error there is a problem with SQLLite?
n
nicholas
06/28/2023, 2:41 AM
Iâm not sure - that error message doesnât give a ton of information, perhaps the server logs show more?
e
Edmondo Porcu
06/28/2023, 2:41 AMI am running this in background on Github Workflows because thatâs what we use for automation, let me see
yes database is locked
n
nicholas
06/28/2023, 3:10 AM
Makes sense - scale will always be difficult with SQLite - if youâre finding thereâs something reproducible that we can improve on here please feel free to open a ticket. Otherwise, Iâd encourage you to take a look at the postgres subchart - obviously this is one of the issues that the hosted Cloud solves out of the box but I understand youâre evaluating that separately.
e
Edmondo Porcu
06/28/2023, 3:20 AMCan I also simply launch the server and use a separate db via env variables or migrations will be difficult?
n
nicholas
06/28/2023, 3:25 AM
Yes - take a look at the docs on hosting with Postgresql for information on how to configure that.\
e
Edmondo Porcu
06/28/2023, 3:31 AM Copy code
File "asyncpg/protocol/protocol.pyx", line 201, in bind_execute
40
asyncpg.exceptions.UniqueViolationError: duplicate key value violates unique constraint "uq_configuration__key"
41
DETAIL: Key (key)=(ENCRYPTION_KEY) already exists.Even using the helm chart, the problem remains
Copy code
prefect.exceptions.PrefectHTTPStatusError: Client error '409 Conflict' for url '<http://localhost:4200/api/block_schemas/>'
Response: {'detail': 'Data integrity conflict. This usually means a unique or foreign key constraint was violated. See server logs for details.'}