Thread
#prefect-server
    Sidd

    Sidd

    1 year ago
    Hi folks, Im running into this weird error where the default prefect tables are not getting created when the server starts up for the first time. I actually got the whole thing working in our staging env, but when I pushed to our prod env, it seems like the default tables don’t exist. I don’t see the tables in prod hasura, like the ones I see in our staging hasura. Any insight/help would be much appreciated!
    Michael Adkins

    Michael Adkins

    1 year ago
    Hi @Sidd -- I'm not sure what would be causing this since you've said your setup works fine in another environment. You can run
    prefect-server database reset -y
    after it spins up to rebuild the tables.
    Note:
    -y
    auto-confirms, you should_NOT_ run this on unbacked up production data
    Sidd

    Sidd

    1 year ago
    Hey @Michael Adkins, thanks for your quick response! This is the first time its been deployed to prod, so there’s no data yet. Do I run that command from my prefect-server container?
    Also, could there be an issue because of the ordering of when each service’s container spins up?
    Michael Adkins

    Michael Adkins

    1 year ago
    Are you deploying on a single node via
    prefect server start
    ?
    Sidd

    Sidd

    1 year ago
    Nope, I’ve deployed it in K8s with each having their own containers
    Michael Adkins

    Michael Adkins

    1 year ago
    Ah okay, I think it should run in any of the containers with the
    prefect-server
    repo instaleld
    Sidd

    Sidd

    1 year ago
    I tried running that reset, but I got this very familiar error:
    Failed safety check: bad 'hasura.host': '<http://prefect-hasura>'
      expected to be one of: 'localhost','hasura','127.0.0.1'
    Aborted!
    Michael Adkins

    Michael Adkins

    1 year ago
    There's a
    --unsafe
    flag to ignore that, basically we don't want to axe a database we shouldn't
    Sidd

    Sidd

    1 year ago
    gotcha! Let me give that a shot
    Michael Adkins

    Michael Adkins

    1 year ago
    So there's checks that it's a typical Prefect Server database setup (which K8s is not)
    We could probably include that url in the checks though
    Sidd

    Sidd

    1 year ago
    I ran that, and got another error:
    Error: HTTPConnectionPool(host='https', port=80): Max retries exceeded with url: //prefect-hasura.prod.demandjump.net:3000/v1/query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f410d04b050>: Failed to establish a new connection: [Errno -2] Name or service not known'))
    Seems like it doesnt like that url
    I believe in our staging environment, I ran into the same issue as well. And I think what I ended up doing was attaching my K8s containers to my localhost ports, and then doing the above. But I was trying to set all this up a couple weeks back and ran into a lot of issues, and was putting out fires, and I think i forgot to document a step on how i got it to work
    Is there a better work around than this?
    Michael Adkins

    Michael Adkins

    1 year ago
    Are you using the Helm chart or rolling your own K8s deployment?
    The GQL pod typically does the db upgrades with https://github.com/PrefectHQ/server/blob/cbf81a70c92507fb351427184a3b5c216326acca/helm/prefect-server/templates/graphql/deployment.yaml#L43 so if it can't run
    prefect-server database
    that may be the root of your problem
    Sidd

    Sidd

    1 year ago
    We’re using our own K8s deployment
    But I did take a look at the above link, and based a lot of our own stuff on it
    Michael Adkins

    Michael Adkins

    1 year ago
    I'm going to have a hard time helping you out then 😕
    Sidd

    Sidd

    1 year ago
    Its actually 90% the same, just without the helm vars
    and pulling ours from vault
    Michael Adkins

    Michael Adkins

    1 year ago
    But there's some small difference that's stopping your database from being setup correctly
    I've never seen that with a server deployment -- pretty much just when developing locally and resetting the database usually fixes it because it's just weird state from development
    Are you running the upgrade command from the GraphQL pod like I linked?
    Can you look at the logs for that init container?
    Sidd

    Sidd

    1 year ago
    Yes, I am running it in that container. I see a similar error message as before
    Running Alembic migrations...
    INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
    INFO  [alembic.runtime.migration] Will assume transactional DDL.
    
    Could not upgrade the database!
    Error: HTTPConnectionPool(host='https', port=80): Max retries exceeded with url: //prefect-hasura.prod.demandjump.net:80/v1/query (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8b1eaab890>: Failed to establish a new connection: [Errno -2] Name or service not known'))
    Not sure why the host is only
    HTTPConnectionPool(host='https'
    and not the whole url
    I can actually do a curl request within that container for the same url and it returns alright
    Michael Adkins

    Michael Adkins

    1 year ago
    Well, that's why your database doesn't have the table.
    What's the value of
    PREFECT_SERVER__HASURA__HOST
    on that container
    Sidd

    Sidd

    1 year ago
    Do I need to remove the
    https
    part?
    Michael Adkins

    Michael Adkins

    1 year ago
    I think so
    The config generates the url for you
    [hasura]
    
    host = "localhost"
    port = 3000
    graphql_url = "http://${hasura.host}:${hasura.port}/v1alpha1/graphql"
    Sidd

    Sidd

    1 year ago
    BOOM! That got it to work!
    Michael Adkins

    Michael Adkins

    1 year ago
    Since you're setting
    host
    with
    http
    it's being templated into the
    graphql_url
    with two https
    Sidd

    Sidd

    1 year ago
    ah! I believe I saw that somewhere when I was trying to get it all working. But thanks a ton @Michael Adkins. Really appreciate the quick help!!
    Michael Adkins

    Michael Adkins

    1 year ago
    No problem 🙂
    Sidd

    Sidd

    1 year ago
    Just FYI, I did have this variable
    PREFECT_SERVER__HASURA__GRAPHQL_URL=<https://prefect-hasura.prod.demandjump.net/v1/graphql>
    So, shouldnt that have overridden the above templated config?
    Michael Adkins

    Michael Adkins

    1 year ago
    That looks to be missing the port
    Although that may not be needed depending on what you setup?
    Sidd

    Sidd

    1 year ago
    yeah, my bad, I think that makes sense! Thanks again!
    Hey @Michael Adkins, I have another issue im running into, if you could help me out. I have a tenant already created and can see it in hasura, but the UI and the agents complain about this:
    Uncaught (in promise) Error: passed invalid or empty tenant object
        at l.setDefaultTenant (index.js:71)
        at vuex.esm.js:840
        at vuex.esm.js:462
        at Array.forEach (<anonymous>)
        at vuex.esm.js:461
        at l._withCommit (vuex.esm.js:620)
        at l.commit (vuex.esm.js:460)
        at l.commit (vuex.esm.js:405)
        at tenantNavGuard.js:12
        at u (runtime.js:63)
    Am I missing some env variable that needs to be set to the tenant or something?
    Michael Adkins

    Michael Adkins

    1 year ago
    There isn't anything to be set to the tenant.. hmm
    What error does the agent give? It should be different than that
    Sidd

    Sidd

    1 year ago
    prefect.utilities.exceptions.ClientError: 400 Client Error: Bad Request for url: <https://prefect-apollo.prod.demandjump.net/graphql>
    The following error messages were provided by the GraphQL server:
        GRAPHQL_VALIDATION_FAILED: Cannot query field "tenant" on type "Query".
    The GraphQL query was:
        query {
                tenant {
                    id
            }
        }
    The passed variables were:
        null
    Michael Adkins

    Michael Adkins

    1 year ago
    It looks like your graphql schema is broken, did you restart apollo/hasura/graphql after fixing the database?
    Sidd

    Sidd

    1 year ago
    It didnt restart apollo, because only hasura and graphql had updates to their env variables. But I deleted all my containers and redeployed them and the UI looks amazing, such a relief lol
    Thank you @Michael Adkins!