https://prefect.io logo
Powered by
Title
s

Slackbot

04/14/2022, 3:16 PM
This message was deleted.
k

Kevin Kho

04/14/2022, 3:22 PM
This looks like the database migrations did not run properly. This may help. Your hasura version may be below the needed one for your server version too
m

Mia

04/14/2022, 3:29 PM
i just tried running database upgrade by running 
kubectl exec prefect-server-graphql-xxx -- prefect-server database upgrade -y
 and it ran successfully but the error persists still
k

Kevin Kho

04/14/2022, 3:31 PM
What is your hasura version?
m

Mia

04/14/2022, 3:35 PM
Image: hasura/graphql-engine:v2.0.9
k

Kevin Kho

04/14/2022, 3:37 PM
that looks right. let me look at some other stuff
did you restart apollo?
m

Mia

04/14/2022, 3:43 PM
yes i did
this is the traceback 
[2022-04-14 15:42:53,225] ERROR - agent | Exception encountered while deploying flow run xxx-xxx-xxx-xxx
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 388, in _deploy_flow_run
    deployment_info = self.deploy_flow(flow_run)
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 430, in deploy_flow
    job_spec = self.generate_job_spec(flow_run=flow_run)
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 542, in generate_job_spec
    flow_run, default=container.get("image")
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/agent.py", line 29, in get_flow_image
    storage = StorageSchema().load(flow_run.flow.storage)
  File "/usr/local/lib/python3.7/site-packages/marshmallow_oneofschema/one_of_schema.py", line 153, in load
    raise exc
marshmallow.exceptions.ValidationError: {'flows': defaultdict(<class 'dict'>, {'my-flow': {'value': ['Field may not be null.']}})}
it looks like it’s not able to get the flow from the storage. I have my flows in GitLab and have stored secrets in job_template.yaml file in s3 bucket and i pass the path in the kubernetes run_config. It worked before so now i’m confused
k

Kevin Kho

04/14/2022, 3:55 PM
I doubt anything would work at the moment because of the version mismatch. Do you mean that if you use something like public Github storage, this would work?
:upvote: 1
m

Mia

04/14/2022, 3:57 PM
which versions should i be checking? I’m using a private GitLab storage and i’ve stored the secrets in kubernetes secret for now and it seems to be available in the agent
on a side note, i did a fresh helm delete perfect-server and helm install prefect-server yesterday so i’m hoping that there’s no version inconsistencies
k

Kevin Kho

04/14/2022, 3:58 PM
if your server is above 1.0, the important thing is just the hasura 2.0. I more frequently see this when hasura is below 2.0.
if you do that there shouldn’t be any inconsistencies
m

Mia

04/14/2022, 4:02 PM
so i guess it’s not the version issue. What else can I check?
k

Kevin Kho

04/14/2022, 4:03 PM
Checking with a team member
a

Anna Geller

04/14/2022, 4:09 PM
You should also check your Postgres version - according to this user, you need Postgres 11, Hasura 2.1.1 to use Prefect 1.2
z

Zanie

04/14/2022, 4:23 PM
Generally this is an issue when Hasura is on version 2.x without the flag that enables 1.x behavior for nulls
🤦‍♂️ 1
https://github.com/PrefectHQ/server/pull/328/files#diff-08e719f6b430038b01759623314c39ef27620a7ed98661ab33d2ded6a937c0c6R60-R61
m

Mia

04/14/2022, 4:39 PM
@Anna Geller It seems like i’m using the right postgresql version -> 
<http://docker.io/bitnami/postgresql:11.9.0-debian-10-r1|docker.io/bitnami/postgresql:11.9.0-debian-10-r1>
 @Zanie do I just pass 
- name: HASURA_GRAPHQL_V1_BOOLEAN_NULL_COLLAPSE
   value: "true"
as an env variable in helm chart Hasura definition?
👍 1
z

Zanie

04/14/2022, 4:54 PM
It should be included in the Helm chart already, perhaps you’re on an older version of the chart?
You can check if it’s missing from the deployment
I think these issues are often from mismatched versions across the chart, but there are a lot of moving parts and it’s hard to say 🙂
:upvote: 1
m

Mia

04/14/2022, 6:49 PM
how do I check which version i’m using for prefect-server? when i run 
kubectl describ pod <prefect agent server pod>
, i only see the image pulled is “latest”. I also checked Hasura graphql is running 2.1.1, postgresql is running 11.9.0, and 
HASURA_GRAPHQL_V1_BOOLEAN_NULL_COLLAPSE
env variable is set “true”. But i’m still getting the same error
i’m using this values.yaml
And the helm chart repo for prefect-server version, i’m using this: 
NAME                    	CHART VERSION	APP VERSION	DESCRIPTION                      
prefecthq/prefect-server	2022.03.29   	2022.03.29 	Prefect Server application bundle
After checking around, we concluded that it’s not the DB version issues but rather server is not able to reach GitLab storage and fetch the flows. We switched to using Module storage and this error was solved.
On a side note, it would be good to get full traceback on the Prefect server UI to understand error messages a bit better. Is there a way to do this? We did set the error level to “DEBUG” but it doesn’t return full traceback
k

Kevin Kho

04/22/2022, 2:26 PM
Oh what? That’s surprising. Yes to the second one. You can raise full traceback by making your own 
@task
decorator. I have an example
import traceback
from prefect import task, Flow
from functools import partial, wraps
from prefect.executors import LocalDaskExecutor

def custom_task(func=None, **task_init_kwargs):
    if func is None:
        return partial(custom_task, **task_init_kwargs)

    @wraps(func)
    def safe_func(**kwargs):
        try:
            return func(**kwargs)
        except Exception as e:
            print(f"Full Traceback: {traceback.format_exc()}")
            raise RuntimeError(type(e)) from None  # from None is necessary to not log the stacktrace

    safe_func.__name__ = func.__name__
    return task(safe_func, **task_init_kwargs)

@custom_task
def abc(x):
    return x

with Flow("custom-decorator-test") as flow:
    abc.map([1,2,3,4,5])

flow.run()

# flow.executor = LocalDaskExecutor(scheduler="processes")

# flow.register("testing")
m

Mia

04/25/2022, 8:30 AM
This is great! Thanks, @Kevin Kho. I’ll give this a try!
3 Views
#prefect-serverJoin Slack
Powered by