https://prefect.io logo
Join Slack
Powered by
This message was deleted.
# prefect-server
s
This message was deleted.
k
This looks like the database migrations did not run properly. This may help. Your hasura version may be below the needed one for your server version too
m
i just tried running database upgrade by running 
kubectl exec prefect-server-graphql-xxx -- prefect-server database upgrade -y
 and it ran successfully but the error persists still
k
What is your hasura version?
m
Image: hasura/graphql-engine:v2.0.9
k
that looks right. let me look at some other stuff
did you restart apollo?
m
yes i did
this is the traceback
Copy code
[2022-04-14 15:42:53,225] ERROR - agent | Exception encountered while deploying flow run xxx-xxx-xxx-xxx
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/agent.py", line 388, in _deploy_flow_run
    deployment_info = self.deploy_flow(flow_run)
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 430, in deploy_flow
    job_spec = self.generate_job_spec(flow_run=flow_run)
  File "/usr/local/lib/python3.7/site-packages/prefect/agent/kubernetes/agent.py", line 542, in generate_job_spec
    flow_run, default=container.get("image")
  File "/usr/local/lib/python3.7/site-packages/prefect/utilities/agent.py", line 29, in get_flow_image
    storage = StorageSchema().load(flow_run.flow.storage)
  File "/usr/local/lib/python3.7/site-packages/marshmallow_oneofschema/one_of_schema.py", line 153, in load
    raise exc
marshmallow.exceptions.ValidationError: {'flows': defaultdict(<class 'dict'>, {'my-flow': {'value': ['Field may not be null.']}})}
it looks like it’s not able to get the flow from the storage. I have my flows in GitLab and have stored secrets in job_template.yaml file in s3 bucket and i pass the path in the kubernetes run_config. It worked before so now i’m confused
k
I doubt anything would work at the moment because of the version mismatch. Do you mean that if you use something like public Github storage, this would work?
upvote 1
m
which versions should i be checking? I’m using a private GitLab storage and i’ve stored the secrets in kubernetes secret for now and it seems to be available in the agent
on a side note, i did a fresh helm delete perfect-server and helm install prefect-server yesterday so i’m hoping that there’s no version inconsistencies
k
if your server is above 1.0, the important thing is just the hasura 2.0. I more frequently see this when hasura is below 2.0.
if you do that there shouldn’t be any inconsistencies
m
so i guess it’s not the version issue. What else can I check?
k
Checking with a team member
a
You should also check your Postgres version - according to this user, you need Postgres 11, Hasura 2.1.1 to use Prefect 1.2
z
Generally this is an issue when Hasura is on version 2.x without the flag that enables 1.x behavior for nulls
🤦‍♂️ 1
m
@Anna Geller It seems like i’m using the right postgresql version -> 
<http://docker.io/bitnami/postgresql:11.9.0-debian-10-r1|docker.io/bitnami/postgresql:11.9.0-debian-10-r1>
 @Zanie do I just pass
Copy code
- name: HASURA_GRAPHQL_V1_BOOLEAN_NULL_COLLAPSE
   value: "true"
as an env variable in helm chart Hasura definition?
👍 1
z
It should be included in the Helm chart already, perhaps you’re on an older version of the chart?
You can check if it’s missing from the deployment
I think these issues are often from mismatched versions across the chart, but there are a lot of moving parts and it’s hard to say 🙂
upvote 1
m
how do I check which version i’m using for prefect-server? when i run 
kubectl describ pod <prefect agent server pod>
, i only see the image pulled is “latest”. I also checked Hasura graphql is running 2.1.1, postgresql is running 11.9.0, and 
HASURA_GRAPHQL_V1_BOOLEAN_NULL_COLLAPSE
env variable is set “true”. But i’m still getting the same error
i’m using this values.yaml
And the helm chart repo for prefect-server version, i’m using this:
Copy code
NAME                    	CHART VERSION	APP VERSION	DESCRIPTION                      
prefecthq/prefect-server	2022.03.29   	2022.03.29 	Prefect Server application bundle
After checking around, we concluded that it’s not the DB version issues but rather server is not able to reach GitLab storage and fetch the flows. We switched to using Module storage and this error was solved.
On a side note, it would be good to get full traceback on the Prefect server UI to understand error messages a bit better. Is there a way to do this? We did set the error level to “DEBUG” but it doesn’t return full traceback
k
Oh what? That’s surprising. Yes to the second one. You can raise full traceback by making your own 
@task
decorator. I have an example
Copy code
import traceback
from prefect import task, Flow
from functools import partial, wraps
from prefect.executors import LocalDaskExecutor

def custom_task(func=None, **task_init_kwargs):
    if func is None:
        return partial(custom_task, **task_init_kwargs)

    @wraps(func)
    def safe_func(**kwargs):
        try:
            return func(**kwargs)
        except Exception as e:
            print(f"Full Traceback: {traceback.format_exc()}")
            raise RuntimeError(type(e)) from None  # from None is necessary to not log the stacktrace

    safe_func.__name__ = func.__name__
    return task(safe_func, **task_init_kwargs)

@custom_task
def abc(x):
    return x

with Flow("custom-decorator-test") as flow:
    abc.map([1,2,3,4,5])

flow.run()

# flow.executor = LocalDaskExecutor(scheduler="processes")

# flow.register("testing")
m
This is great! Thanks, @Kevin Kho. I’ll give this a try!
5 Views
Open in Slack
PreviousNext
Powered by