https://prefect.io logo
Join Slack
Powered by
Hi folks, Im running into this weird error where t...
# prefect-server
s
Hi folks, Im running into this weird error where the default prefect tables are not getting created when the server starts up for the first time. I actually got the whole thing working in our staging env, but when I pushed to our prod env, it seems like the default tables don’t exist. I don’t see the tables in prod hasura, like the ones I see in our staging hasura. Any insight/help would be much appreciated!
z
Hi @Sidd -- I'm not sure what would be causing this since you've said your setup works fine in another environment. You can run 
prefect-server database reset -y
after it spins up to rebuild the tables.
Note: 
-y
auto-confirms, you should_NOT_ run this on unbacked up production data
s
Hey @Zanie, thanks for your quick response! This is the first time its been deployed to prod, so there’s no data yet. Do I run that command from my prefect-server container?
Also, could there be an issue because of the ordering of when each service’s container spins up?
z
Are you deploying on a single node via 
prefect server start
?
s
Nope, I’ve deployed it in K8s with each having their own containers
z
Ah okay, I think it should run in any of the containers with the 
prefect-server
repo instaleld
s
I tried running that reset, but I got this very familiar error:
Copy code
Failed safety check: bad 'hasura.host': '<http://prefect-hasura>'
  expected to be one of: 'localhost','hasura','127.0.0.1'
Aborted!
z
There's a 
--unsafe
flag to ignore that, basically we don't want to axe a database we shouldn't
s
gotcha! Let me give that a shot
z
So there's checks that it's a typical Prefect Server database setup (which K8s is not)
We could probably include that url in the checks though
s
I ran that, and got another error:
Copy code
Error: HTTPConnectionPool(host='https', port=80): Max retries exceeded with url: //prefect-hasura.prod.demandjump.net:3000/v1/query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f410d04b050>: Failed to establish a new connection: [Errno -2] Name or service not known'))
Seems like it doesnt like that url
I believe in our staging environment, I ran into the same issue as well. And I think what I ended up doing was attaching my K8s containers to my localhost ports, and then doing the above. But I was trying to set all this up a couple weeks back and ran into a lot of issues, and was putting out fires, and I think i forgot to document a step on how i got it to work
Is there a better work around than this?
z
Are you using the Helm chart or rolling your own K8s deployment?
The GQL pod typically does the db upgrades with https://github.com/PrefectHQ/server/blob/cbf81a70c92507fb351427184a3b5c216326acca/helm/prefect-server/templates/graphql/deployment.yaml#L43 so if it can't run 
prefect-server database
that may be the root of your problem
s
We’re using our own K8s deployment
But I did take a look at the above link, and based a lot of our own stuff on it
z
I'm going to have a hard time helping you out then 😕
s
Its actually 90% the same, just without the helm vars
and pulling ours from vault
z
But there's some small difference that's stopping your database from being setup correctly
I've never seen that with a server deployment -- pretty much just when developing locally and resetting the database usually fixes it because it's just weird state from development
Are you running the upgrade command from the GraphQL pod like I linked?
Can you look at the logs for that init container?
s
Yes, I am running it in that container. I see a similar error message as before
Copy code
Running Alembic migrations...
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.

Could not upgrade the database!
Error: HTTPConnectionPool(host='https', port=80): Max retries exceeded with url: //prefect-hasura.prod.demandjump.net:80/v1/query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8b1eaab890>: Failed to establish a new connection: [Errno -2] Name or service not known'))
Not sure why the host is only 
HTTPConnectionPool(host='https'
and not the whole url
I can actually do a curl request within that container for the same url and it returns alright
z
Well, that's why your database doesn't have the table.
What's the value of 
PREFECT_SERVER__HASURA__HOST
on that container
s
Do I need to remove the 
https
part?
z
I think so
The config generates the url for you
Copy code
[hasura]

host = "localhost"
port = 3000
graphql_url = "http://${hasura.host}:${hasura.port}/v1alpha1/graphql"
s
BOOM! That got it to work!
z
Since you're setting 
host
with 
http
it's being templated into the 
graphql_url
with two `http`s
s
ah! I believe I saw that somewhere when I was trying to get it all working. But thanks a ton @Zanie. Really appreciate the quick help!!
z
No problem 🙂
s
Just FYI, I did have this variable 
PREFECT_SERVER__HASURA__GRAPHQL_URL=<https://prefect-hasura.prod.demandjump.net/v1/graphql>
So, shouldnt that have overridden the above templated config?
z
That looks to be missing the port
Although that may not be needed depending on what you setup?
s
yeah, my bad, I think that makes sense! Thanks again!
Hey @Zanie, I have another issue im running into, if you could help me out. I have a tenant already created and can see it in hasura, but the UI and the agents complain about this:
Copy code
Uncaught (in promise) Error: passed invalid or empty tenant object
    at l.setDefaultTenant (index.js:71)
    at vuex.esm.js:840
    at vuex.esm.js:462
    at Array.forEach (<anonymous>)
    at vuex.esm.js:461
    at l._withCommit (vuex.esm.js:620)
    at l.commit (vuex.esm.js:460)
    at l.commit (vuex.esm.js:405)
    at tenantNavGuard.js:12
    at u (runtime.js:63)
Am I missing some env variable that needs to be set to the tenant or something?
z
There isn't anything to be set to the tenant.. hmm
What error does the agent give? It should be different than that
s
Copy code
prefect.utilities.exceptions.ClientError: 400 Client Error: Bad Request for url: <https://prefect-apollo.prod.demandjump.net/graphql>
The following error messages were provided by the GraphQL server:
    GRAPHQL_VALIDATION_FAILED: Cannot query field "tenant" on type "Query".
The GraphQL query was:
    query {
            tenant {
                id
        }
    }
The passed variables were:
    null
z
It looks like your graphql schema is broken, did you restart apollo/hasura/graphql after fixing the database?
s
It didnt restart apollo, because only hasura and graphql had updates to their env variables. But I deleted all my containers and redeployed them and the UI looks amazing, such a relief lol
Thank you @Zanie!
Open in Slack
PreviousNext
Powered by