https://prefect.io logo
Powered by
Title
s

Sidd

04/02/2021, 5:06 PM
Hi folks, Im running into this weird error where the default prefect tables are not getting created when the server starts up for the first time. I actually got the whole thing working in our staging env, but when I pushed to our prod env, it seems like the default tables don’t exist. I don’t see the tables in prod hasura, like the ones I see in our staging hasura. Any insight/help would be much appreciated!
z

Zanie

04/02/2021, 5:17 PM
Hi @Sidd -- I'm not sure what would be causing this since you've said your setup works fine in another environment. You can run 
prefect-server database reset -y
after it spins up to rebuild the tables.
Note: 
-y
auto-confirms, you should_NOT_ run this on unbacked up production data
s

Sidd

04/02/2021, 5:19 PM
Hey @Zanie, thanks for your quick response! This is the first time its been deployed to prod, so there’s no data yet. Do I run that command from my prefect-server container?
Also, could there be an issue because of the ordering of when each service’s container spins up?
z

Zanie

04/02/2021, 5:20 PM
Are you deploying on a single node via 
prefect server start
?
s

Sidd

04/02/2021, 5:24 PM
Nope, I’ve deployed it in K8s with each having their own containers
z

Zanie

04/02/2021, 5:25 PM
Ah okay, I think it should run in any of the containers with the 
prefect-server
repo instaleld
s

Sidd

04/02/2021, 5:25 PM
I tried running that reset, but I got this very familiar error: 
Failed safety check: bad 'hasura.host': '<http://prefect-hasura>'
  expected to be one of: 'localhost','hasura','127.0.0.1'
Aborted!
z

Zanie

04/02/2021, 5:26 PM
There's a 
--unsafe
flag to ignore that, basically we don't want to axe a database we shouldn't
s

Sidd

04/02/2021, 5:26 PM
gotcha! Let me give that a shot
z

Zanie

04/02/2021, 5:26 PM
So there's checks that it's a typical Prefect Server database setup (which K8s is not)
We could probably include that url in the checks though
s

Sidd

04/02/2021, 5:34 PM
I ran that, and got another error: 
Error: HTTPConnectionPool(host='https', port=80): Max retries exceeded with url: //prefect-hasura.prod.demandjump.net:3000/v1/query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f410d04b050>: Failed to establish a new connection: [Errno -2] Name or service not known'))
Seems like it doesnt like that url
I believe in our staging environment, I ran into the same issue as well. And I think what I ended up doing was attaching my K8s containers to my localhost ports, and then doing the above. But I was trying to set all this up a couple weeks back and ran into a lot of issues, and was putting out fires, and I think i forgot to document a step on how i got it to work
Is there a better work around than this?
z

Zanie

04/02/2021, 5:40 PM
Are you using the Helm chart or rolling your own K8s deployment?
The GQL pod typically does the db upgrades with https://github.com/PrefectHQ/server/blob/cbf81a70c92507fb351427184a3b5c216326acca/helm/prefect-server/templates/graphql/deployment.yaml#L43 so if it can't run 
prefect-server database
that may be the root of your problem
s

Sidd

04/02/2021, 5:43 PM
We’re using our own K8s deployment
But I did take a look at the above link, and based a lot of our own stuff on it
z

Zanie

04/02/2021, 5:45 PM
I'm going to have a hard time helping you out then 😕
s

Sidd

04/02/2021, 5:46 PM
Its actually 90% the same, just without the helm vars
and pulling ours from vault
z

Zanie

04/02/2021, 5:47 PM
But there's some small difference that's stopping your database from being setup correctly
I've never seen that with a server deployment -- pretty much just when developing locally and resetting the database usually fixes it because it's just weird state from development
Are you running the upgrade command from the GraphQL pod like I linked?
Can you look at the logs for that init container?
s

Sidd

04/02/2021, 5:50 PM
Yes, I am running it in that container. I see a similar error message as before
Running Alembic migrations...
INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO  [alembic.runtime.migration] Will assume transactional DDL.

Could not upgrade the database!
Error: HTTPConnectionPool(host='https', port=80): Max retries exceeded with url: //prefect-hasura.prod.demandjump.net:80/v1/query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8b1eaab890>: Failed to establish a new connection: [Errno -2] Name or service not known'))
Not sure why the host is only 
HTTPConnectionPool(host='https'
and not the whole url
I can actually do a curl request within that container for the same url and it returns alright
z

Zanie

04/02/2021, 5:51 PM
Well, that's why your database doesn't have the table.
What's the value of 
PREFECT_SERVER__HASURA__HOST
on that container
s

Sidd

04/02/2021, 5:51 PM
its the following: https://prefect-hasura.prod.demandjump.net
Do I need to remove the 
https
part?
z

Zanie

04/02/2021, 5:52 PM
I think so
The config generates the url for you
[hasura]

host = "localhost"
port = 3000
graphql_url = "http://${hasura.host}:${hasura.port}/v1alpha1/graphql"
s

Sidd

04/02/2021, 5:53 PM
BOOM! That got it to work!
z

Zanie

04/02/2021, 5:53 PM
Since you're setting 
host
with 
http
it's being templated into the 
graphql_url
with two `http`s
s

Sidd

04/02/2021, 5:54 PM
ah! I believe I saw that somewhere when I was trying to get it all working. But thanks a ton @Zanie. Really appreciate the quick help!!
z

Zanie

04/02/2021, 5:54 PM
No problem 🙂
s

Sidd

04/02/2021, 5:56 PM
Just FYI, I did have this variable 
PREFECT_SERVER__HASURA__GRAPHQL_URL=<https://prefect-hasura.prod.demandjump.net/v1/graphql>
So, shouldnt that have overridden the above templated config?
z

Zanie

04/02/2021, 6:05 PM
That looks to be missing the port
Although that may not be needed depending on what you setup?
s

Sidd

04/02/2021, 6:07 PM
yeah, my bad, I think that makes sense! Thanks again!
Hey @Zanie, I have another issue im running into, if you could help me out. I have a tenant already created and can see it in hasura, but the UI and the agents complain about this: 
Uncaught (in promise) Error: passed invalid or empty tenant object
    at l.setDefaultTenant (index.js:71)
    at vuex.esm.js:840
    at vuex.esm.js:462
    at Array.forEach (<anonymous>)
    at vuex.esm.js:461
    at l._withCommit (vuex.esm.js:620)
    at l.commit (vuex.esm.js:460)
    at l.commit (vuex.esm.js:405)
    at tenantNavGuard.js:12
    at u (runtime.js:63)
Am I missing some env variable that needs to be set to the tenant or something?
z

Zanie

04/02/2021, 6:24 PM
There isn't anything to be set to the tenant.. hmm
What error does the agent give? It should be different than that
s

Sidd

04/02/2021, 6:25 PM
prefect.utilities.exceptions.ClientError: 400 Client Error: Bad Request for url: <https://prefect-apollo.prod.demandjump.net/graphql>
The following error messages were provided by the GraphQL server:
    GRAPHQL_VALIDATION_FAILED: Cannot query field "tenant" on type "Query".
The GraphQL query was:
    query {
            tenant {
                id
        }
    }
The passed variables were:
    null
z

Zanie

04/02/2021, 6:29 PM
It looks like your graphql schema is broken, did you restart apollo/hasura/graphql after fixing the database?
s

Sidd

04/02/2021, 6:30 PM
It didnt restart apollo, because only hasura and graphql had updates to their env variables. But I deleted all my containers and redeployed them and the UI looks amazing, such a relief lol
Thank you @Zanie!
#prefect-serverJoin Slack
Powered by