s

    Slackbot

    5 months ago
    This message was deleted.
    Kevin Kho

    Kevin Kho

    5 months ago
    This looks like the database migrations did not run properly. This may help. Your hasura version may be below the needed one for your server version too
    m

    Maria

    5 months ago
    i just tried running database upgrade by running
    kubectl exec prefect-server-graphql-xxx -- prefect-server database upgrade -y
    and it ran successfully but the error persists still
    Kevin Kho

    Kevin Kho

    5 months ago
    What is your hasura version?
    m

    Maria

    5 months ago
    Image: hasura/graphql-engine:v2.0.9
    Kevin Kho

    Kevin Kho

    5 months ago
    that looks right. let me look at some other stuff
    did you restart apollo?
    m

    Maria

    5 months ago
    yes i did
    this is the traceback
    [2022-04-14 15:42:53,225] ERROR - agent | Exception encountered while deploying flow run xxx-xxx-xxx-xxx
    Traceback (most recent call last):
      File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 388, in _deploy_flow_run
        deployment_info = self.deploy_flow(flow_run)
      File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 430, in deploy_flow
        job_spec = self.generate_job_spec(flow_run=flow_run)
      File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 542, in generate_job_spec
        flow_run, default=container.get("image")
      File "/usr/local/lib/python3.7/site-packages/prefect/utilities/agent.py", line 29, in get_flow_image
        storage = StorageSchema().load(flow_run.flow.storage)
      File "/usr/local/lib/python3.7/site-packages/marshmallow_oneofschema/one_of_schema.py", line 153, in load
        raise exc
    marshmallow.exceptions.ValidationError: {'flows': defaultdict(<class 'dict'>, {'my-flow': {'value': ['Field may not be null.']}})}
    it looks like it’s not able to get the flow from the storage. I have my flows in GitLab and have stored secrets in job_template.yaml file in s3 bucket and i pass the path in the kubernetes run_config. It worked before so now i’m confused
    Kevin Kho

    Kevin Kho

    5 months ago
    I doubt anything would work at the moment because of the version mismatch. Do you mean that if you use something like public Github storage, this would work?
    m

    Maria

    5 months ago
    which versions should i be checking? I’m using a private GitLab storage and i’ve stored the secrets in kubernetes secret for now and it seems to be available in the agent
    on a side note, i did a fresh helm delete perfect-server and helm install prefect-server yesterday so i’m hoping that there’s no version inconsistencies
    Kevin Kho

    Kevin Kho

    5 months ago
    if your server is above 1.0, the important thing is just the hasura 2.0. I more frequently see this when hasura is below 2.0.
    if you do that there shouldn’t be any inconsistencies
    m

    Maria

    5 months ago
    so i guess it’s not the version issue. What else can I check?
    Kevin Kho

    Kevin Kho

    5 months ago
    Checking with a team member
    Anna Geller

    Anna Geller

    5 months ago
    You should also check your Postgres version - according to this user, you need Postgres 11, Hasura 2.1.1 to use Prefect 1.2
    Michael Adkins

    Michael Adkins

    5 months ago
    Generally this is an issue when Hasura is on version 2.x without the flag that enables 1.x behavior for nulls
    m

    Maria

    5 months ago
    @Anna Geller It seems like i’m using the right postgresql version ->
    <http://docker.io/bitnami/postgresql:11.9.0-debian-10-r1|docker.io/bitnami/postgresql:11.9.0-debian-10-r1>
    @Michael Adkins do I just pass
    - name: HASURA_GRAPHQL_V1_BOOLEAN_NULL_COLLAPSE
       value: "true"
    as an env variable in helm chart Hasura definition?
    Michael Adkins

    Michael Adkins

    5 months ago
    It should be included in the Helm chart already, perhaps you’re on an older version of the chart?
    You can check if it’s missing from the deployment
    I think these issues are often from mismatched versions across the chart, but there are a lot of moving parts and it’s hard to say 🙂
    m

    Maria

    5 months ago
    how do I check which version i’m using for prefect-server? when i run
    kubectl describ pod <prefect agent server pod>
    , i only see the image pulled is “latest”. I also checked Hasura graphql is running 2.1.1, postgresql is running 11.9.0, and
    HASURA_GRAPHQL_V1_BOOLEAN_NULL_COLLAPSE
    env variable is set “true”. But i’m still getting the same error
    i’m using this values.yaml
    And the helm chart repo for prefect-server version, i’m using this:
    NAME                    	CHART VERSION	APP VERSION	DESCRIPTION                      
    prefecthq/prefect-server	2022.03.29   	2022.03.29 	Prefect Server application bundle
    After checking around, we concluded that it’s not the DB version issues but rather server is not able to reach GitLab storage and fetch the flows. We switched to using Module storage and this error was solved.
    On a side note, it would be good to get full traceback on the Prefect server UI to understand error messages a bit better. Is there a way to do this? We did set the error level to “DEBUG” but it doesn’t return full traceback
    Kevin Kho

    Kevin Kho

    5 months ago
    Oh what? That’s surprising. Yes to the second one. You can raise full traceback by making your own
    @task
    decorator. I have an example
    import traceback
    from prefect import task, Flow
    from functools import partial, wraps
    from prefect.executors import LocalDaskExecutor
    
    def custom_task(func=None, **task_init_kwargs):
        if func is None:
            return partial(custom_task, **task_init_kwargs)
    
        @wraps(func)
        def safe_func(**kwargs):
            try:
                return func(**kwargs)
            except Exception as e:
                print(f"Full Traceback: {traceback.format_exc()}")
                raise RuntimeError(type(e)) from None  # from None is necessary to not log the stacktrace
    
        safe_func.__name__ = func.__name__
        return task(safe_func, **task_init_kwargs)
    
    @custom_task
    def abc(x):
        return x
    
    with Flow("custom-decorator-test") as flow:
        abc.map([1,2,3,4,5])
    
    flow.run()
    
    # flow.executor = LocalDaskExecutor(scheduler="processes")
    
    # flow.register("testing")
    m

    Maria

    5 months ago
    This is great! Thanks, @Kevin Kho. I’ll give this a try!