https://prefect.io
Join Slack
Hello world. Does the cloud edition capture identity information with the flow runs?
# prefect-cloud
e
Hello world. Does the cloud edition capture identity information with the flow runs?
n
Hi @Edmondo Porcu - do you mean information about who created runs?
e
yes
given the open source version has no authentication
n
Got it - yes, Cloud tracks captures that information in a few ways; the actor (user or service account) responsible for creating a given flow run is referenced in flow run details (this is visible in the app). For organizations, this information is also captured in the form of audit logs.
e
by the way, do you know that the recent docker images do not work if you do 
docker run prefect:whatever prefect server start
they throw an exception? 😄
n
That doesn’t sound good - please open a ticket in the prefect repo with some info about what you’re trying, including some system info if you’re able so we can triage that appropriately
e
Thanks. Maybe you do have a better workaround for our problems. We are evaluating prefect using hte Helm Chart before purchasing the cloud version (we do have a quote already) We have to run a production job, but we haven’t got prefect deployed in production kubernetes yet, so we want to run it “serverless”. However, if we run it serverless, it seems kind of impossible to submit in parallel ten flows to databricks: • There is a locking error on the SQLLite • they do not proceed in parallel As a workaround I am trying to start a local prefect server within a Github Workflows
btw, I finally managed to start prefect on Github Workflows, but now I get the following error:
Copy code
02:33:17.023 | INFO    | Flow run 'attentive-wildcat' - View at <http://localhost:4200/flow-runs/flow-run/f01d5f08-1302-440b-8263-10a93bdbc572>
02:33:17.218 | ERROR   | Flow run 'vivid-donkey' - Crash detected! Execution was interrupted by an unexpected exception: PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<http://localhost:4200/api/block_documents/>'
Response: {'exception_message': 'Internal Server Error'}
For more information check: <https://httpstatuses.com/500>

02:33:17.244 | ERROR   | Flow run 'big-vole' - Crash detected! Execution was interrupted by an unexpected exception: PrefectHTTPStatusError: Server error '500 Internal Server Error' for url '<http://localhost:4200/api/block_documents/>'
Response: {'exception_message': 'Internal Server Error'}
For more information check: <https://httpstatuses.com/500>
n
It sounds like you might be hitting the limit of what a database like SQLite can offer - as mentioned in the docs, production use cases (or just ones with heavy parallelism) will usually require Postgres. The helm chart contains a subchart to help with this
e
Yeah I am now starting the local deployment with 
prefect server start
However I am not using blocks, and I have no idea why there is that error
It sounds extraordinarily complicated given that we need to kick 10 databricks jobs in parallel
Do you think underlying the 500 error there is a problem with SQLLite?
n
I’m not sure - that error message doesn’t give a ton of information, perhaps the server logs show more?
e
I am running this in background on Github Workflows because that’s what we use for automation, let me see
yes database is locked
n
Makes sense - scale will always be difficult with SQLite - if you’re finding there’s something reproducible that we can improve on here please feel free to open a ticket. Otherwise, I’d encourage you to take a look at the postgres subchart - obviously this is one of the issues that the hosted Cloud solves out of the box but I understand you’re evaluating that separately.
e
Can I also simply launch the server and use a separate db via env variables or migrations will be difficult?
n
Yes - take a look at the docs on hosting with Postgresql for information on how to configure that.\
e
Copy code
File "asyncpg/protocol/protocol.pyx", line 201, in bind_execute
40
asyncpg.exceptions.UniqueViolationError: duplicate key value violates unique constraint "uq_configuration__key"
41
DETAIL:  Key (key)=(ENCRYPTION_KEY) already exists.
Even using the helm chart, the problem remains
Copy code
prefect.exceptions.PrefectHTTPStatusError: Client error '409 Conflict' for url '<http://localhost:4200/api/block_schemas/>'
Response: {'detail': 'Data integrity conflict. This usually means a unique or foreign key constraint was violated. See server logs for details.'}
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by