Hi folks, Im running into this weird error where t...
# prefect-server
Hi folks, Im running into this weird error where the default prefect tables are not getting created when the server starts up for the first time. I actually got the whole thing working in our staging env, but when I pushed to our prod env, it seems like the default tables don’t exist. I don’t see the tables in prod hasura, like the ones I see in our staging hasura. Any insight/help would be much appreciated!
Hi @Sidd -- I'm not sure what would be causing this since you've said your setup works fine in another environment. You can run
prefect-server database reset -y
after it spins up to rebuild the tables.
auto-confirms, you should_NOT_ run this on unbacked up production data
Hey @Zanie, thanks for your quick response! This is the first time its been deployed to prod, so there’s no data yet. Do I run that command from my prefect-server container?
Also, could there be an issue because of the ordering of when each service’s container spins up?
Are you deploying on a single node via
prefect server start
Nope, I’ve deployed it in K8s with each having their own containers
Ah okay, I think it should run in any of the containers with the
repo instaleld
I tried running that reset, but I got this very familiar error:
Copy code
Failed safety check: bad 'hasura.host': '<http://prefect-hasura>'
  expected to be one of: 'localhost','hasura',''
There's a
flag to ignore that, basically we don't want to axe a database we shouldn't
gotcha! Let me give that a shot
So there's checks that it's a typical Prefect Server database setup (which K8s is not)
We could probably include that url in the checks though
I ran that, and got another error:
Copy code
Error: HTTPConnectionPool(host='https', port=80): Max retries exceeded with url: //prefect-hasura.prod.demandjump.net:3000/v1/query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f410d04b050>: Failed to establish a new connection: [Errno -2] Name or service not known'))
Seems like it doesnt like that url
I believe in our staging environment, I ran into the same issue as well. And I think what I ended up doing was attaching my K8s containers to my localhost ports, and then doing the above. But I was trying to set all this up a couple weeks back and ran into a lot of issues, and was putting out fires, and I think i forgot to document a step on how i got it to work
Is there a better work around than this?
Are you using the Helm chart or rolling your own K8s deployment?
The GQL pod typically does the db upgrades with https://github.com/PrefectHQ/server/blob/cbf81a70c92507fb351427184a3b5c216326acca/helm/prefect-server/templates/graphql/deployment.yaml#L43 so if it can't run
prefect-server database
that may be the root of your problem
We’re using our own K8s deployment
But I did take a look at the above link, and based a lot of our own stuff on it
I'm going to have a hard time helping you out then 😕
Its actually 90% the same, just without the helm vars
and pulling ours from vault
But there's some small difference that's stopping your database from being setup correctly
I've never seen that with a server deployment -- pretty much just when developing locally and resetting the database usually fixes it because it's just weird state from development
Are you running the upgrade command from the GraphQL pod like I linked?
Can you look at the logs for that init container?
Yes, I am running it in that container. I see a similar error message as before
Copy code
Running Alembic migrations...
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.

Could not upgrade the database!
Error: HTTPConnectionPool(host='https', port=80): Max retries exceeded with url: //prefect-hasura.prod.demandjump.net:80/v1/query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8b1eaab890>: Failed to establish a new connection: [Errno -2] Name or service not known'))
Not sure why the host is only
and not the whole url
I can actually do a curl request within that container for the same url and it returns alright
Well, that's why your database doesn't have the table.
What's the value of
on that container
Do I need to remove the
I think so
The config generates the url for you
Copy code

host = "localhost"
port = 3000
graphql_url = "http://${hasura.host}:${hasura.port}/v1alpha1/graphql"
BOOM! That got it to work!
Since you're setting
it's being templated into the
with two `http`s
ah! I believe I saw that somewhere when I was trying to get it all working. But thanks a ton @Zanie. Really appreciate the quick help!!
No problem 🙂
Just FYI, I did have this variable
So, shouldnt that have overridden the above templated config?
That looks to be missing the port
Although that may not be needed depending on what you setup?
yeah, my bad, I think that makes sense! Thanks again!
Hey @Zanie, I have another issue im running into, if you could help me out. I have a tenant already created and can see it in hasura, but the UI and the agents complain about this:
Copy code
Uncaught (in promise) Error: passed invalid or empty tenant object
    at l.setDefaultTenant (index.js:71)
    at vuex.esm.js:840
    at vuex.esm.js:462
    at Array.forEach (<anonymous>)
    at vuex.esm.js:461
    at l._withCommit (vuex.esm.js:620)
    at l.commit (vuex.esm.js:460)
    at l.commit (vuex.esm.js:405)
    at tenantNavGuard.js:12
    at u (runtime.js:63)
Am I missing some env variable that needs to be set to the tenant or something?
There isn't anything to be set to the tenant.. hmm
What error does the agent give? It should be different than that
Copy code
prefect.utilities.exceptions.ClientError: 400 Client Error: Bad Request for url: <https://prefect-apollo.prod.demandjump.net/graphql>
The following error messages were provided by the GraphQL server:
    GRAPHQL_VALIDATION_FAILED: Cannot query field "tenant" on type "Query".
The GraphQL query was:
    query {
            tenant {
The passed variables were:
It looks like your graphql schema is broken, did you restart apollo/hasura/graphql after fixing the database?
It didnt restart apollo, because only hasura and graphql had updates to their env variables. But I deleted all my containers and redeployed them and the UI looks amazing, such a relief lol
Thank you @Zanie!