i'm using a small dask cluster, server flavor, v0....
# prefect-community
s
i'm using a small dask cluster, server flavor, v0.13.9, localagent. After upgrade to v0.13.10 I cannot start a flow. The agent prints
Copy code
[2020-10-07 14:26:37,427] ERROR - agent1 | 400 Client Error: Bad Request for url: <http://localhost:4200/>
But it is still possible to register flows as usual. I seem them in the UI, too. I switched back to v0.13.9 and the error is gone. I only changed the Prefect version -- no configuration and no flow code. I'd like to provide you with more information about this bug but the agent is ignoring
$PREFECT__LOGGING__LEVEL=DEBUG
it seems. Is there any change in the enviroment for the agent? Why can I register flows but not start them?
j
Hi @Sven Teresniak there was a change that recently went out in the
0.13.10
server docker images that fixes this and you may need to restart your server 🙂
s
okay, maybe I'm not using the latest version of that image with tag 0.13.10. switching to
imagePullPolicy: Always
testing…
works! damn stupid k8s. should use this policy by default. 😞 thanks a lot, @josh 🙂
🎉 1
j
kubernetes does what kubernetes does because kubernetes is kubernetes
r
I have the same error with 0.13.10 on my local machine while using Prefect’s Cloud API. Bad Request, HTTP400 with body:
{"errors":[{"message":"Cannot query field \\"run_config\\" on type \\"flow\\".","extensions":{"code":"GRAPHQL_VALIDATION_FAILED"}}]}
It’s happening AFTER the Agent has found a Flow for Execution. Is this the same error as you had?
j
@Raphaël Riel looking into this 🤔
@Raphaël Riel I have found the issue (an image needs to be updated to account for this new field) stand by while I get it updated. In the meantime downgrading to 0.13.9 should fix it 🙂
r
I’m in the process of Downgrading! I’ll keep you posted!
Thanks.
While the subject is started @josh, I would like to suggest that Exception being logged using
logger.exception
or by filling the
exc_info
param. As with https://github.com/PrefectHQ/prefect/blob/0.13.10/src/prefect/agent/agent.py#L476 This way I would have gotten the whole exception instead of a simple
400 Client Error: Bad Request for url: <https://api.prefect.io/>
in my logs.
WDYT? Need/Want an Issue on GitHub?
j
I don’t actually think updating that line will provide any more information because that error is returned from Apollo in the backend when it receives a malformed GraphQL request
Would you mind actually opening an issue for this on the server repo https://github.com/PrefectHQ/server because it may be a setting we can expose on Apollo!
r
The text response for the Bad Request returned is what’s put in the `ClientError'`s Message. Getting access to that Exception’s message was my first step in trying to figure out what was going on (Usually HTTP 400 are Client’s Errors, so I thought this was on my side!)
Is the pictured Error’s text is what you want to expose in Apollo?
j
Oh interesting if it is actually responding with that information then we should definitely raise it! Either an issue or PR would be great 🙂
Also Cloud has been updated with the appropriate fields and you should be good to go back to
0.13.10
!
r
Alright, I’ll open an Issue and check if I can pitch in for a PR! Trying to rollback to 0.13.10 at the moment.
All good for 0.13.10
👍 1