https://prefect.io logo
e

Edmondo Porcu

06/28/2023, 12:24 AM
Hello world. Does the cloud edition capture identity information with the flow runs?
n

nicholas

06/28/2023, 12:30 AM
Hi @Edmondo Porcu - do you mean information about who created runs?
e

Edmondo Porcu

06/28/2023, 12:43 AM
yes
given the open source version has no authentication
n

nicholas

06/28/2023, 1:16 AM
Got it - yes, Cloud tracks captures that information in a few ways; the actor (user or service account) responsible for creating a given flow run is referenced in flow run details (this is visible in the app). For organizations, this information is also captured in the form of audit logs.
e

Edmondo Porcu

06/28/2023, 2:24 AM
by the way, do you know that the recent docker images do not work if you do
docker run prefect:whatever prefect server start
they throw an exception? 😄
n

nicholas

06/28/2023, 2:25 AM
That doesn’t sound good - please open a ticket in the prefect repo with some info about what you’re trying, including some system info if you’re able so we can triage that appropriately
e

Edmondo Porcu

06/28/2023, 2:28 AM
Thanks. Maybe you do have a better workaround for our problems. We are evaluating prefect using hte Helm Chart before purchasing the cloud version (we do have a quote already) We have to run a production job, but we haven’t got prefect deployed in production kubernetes yet, so we want to run it “serverless”. However, if we run it serverless, it seems kind of impossible to submit in parallel ten flows to databricks: • There is a locking error on the SQLLite • they do not proceed in parallel As a workaround I am trying to start a local prefect server within a Github Workflows
btw, I finally managed to start prefect on Github Workflows, but now I get the following error:
Copy code
02:33:17.023 | INFO    | Flow run 'attentive-wildcat' - View at <http://localhost:4200/flow-runs/flow-run/f01d5f08-1302-440b-8263-10a93bdbc572>
02:33:17.218 | ERROR   | Flow run 'vivid-donkey' - Crash detected! Execution was interrupted by an unexpected exception: PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<http://localhost:4200/api/block_documents/>'
Response: {'exception_message': 'Internal Server Error'}
For more information check: <https://httpstatuses.com/500>

02:33:17.244 | ERROR   | Flow run 'big-vole' - Crash detected! Execution was interrupted by an unexpected exception: PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<http://localhost:4200/api/block_documents/>'
Response: {'exception_message': 'Internal Server Error'}
For more information check: <https://httpstatuses.com/500>
n

nicholas

06/28/2023, 2:36 AM
It sounds like you might be hitting the limit of what a database like SQLite can offer - as mentioned in the docs, production use cases (or just ones with heavy parallelism) will usually require Postgres. The helm chart contains a subchart to help with this
e

Edmondo Porcu

06/28/2023, 2:36 AM
Yeah I am now starting the local deployment with
prefect server start
However I am not using blocks, and I have no idea why there is that error
It sounds extraordinarily complicated given that we need to kick 10 databricks jobs in parallel
Do you think underlying the 500 error there is a problem with SQLLite?
n

nicholas

06/28/2023, 2:41 AM
I’m not sure - that error message doesn’t give a ton of information, perhaps the server logs show more?
e

Edmondo Porcu

06/28/2023, 2:41 AM
I am running this in background on Github Workflows because that’s what we use for automation, let me see
yes database is locked
n

nicholas

06/28/2023, 3:10 AM
Makes sense - scale will always be difficult with SQLite - if you’re finding there’s something reproducible that we can improve on here please feel free to open a ticket. Otherwise, I’d encourage you to take a look at the postgres subchart - obviously this is one of the issues that the hosted Cloud solves out of the box but I understand you’re evaluating that separately.
e

Edmondo Porcu

06/28/2023, 3:20 AM
Can I also simply launch the server and use a separate db via env variables or migrations will be difficult?
n

nicholas

06/28/2023, 3:25 AM
Yes - take a look at the docs on hosting with Postgresql for information on how to configure that.\
e

Edmondo Porcu

06/28/2023, 3:31 AM
Copy code
File "asyncpg/protocol/protocol.pyx", line 201, in bind_execute
40
asyncpg.exceptions.UniqueViolationError: duplicate key value violates unique constraint "uq_configuration__key"
41
DETAIL:  Key (key)=(ENCRYPTION_KEY) already exists.
Even using the helm chart, the problem remains
Copy code
prefect.exceptions.PrefectHTTPStatusError: Client error '409 Conflict' for url '<http://localhost:4200/api/block_schemas/>'
Response: {'detail': 'Data integrity conflict. This usually means a unique or foreign key constraint was violated. See server logs for details.'}