Roey Brecher

11/25/2020, 2:19 PM
Hello. I’d like to raise the issue of changes in recent prefect versions, that break our deployment. It seems that if Prefect core + server + docker image versions aren’t synchronised to the exact patch version (e.g 0.13.13 might not work with 0.13.17), things might break down. Usually “patch” versions should not include any breaking change. In order for us to make sure our production flows won’t break, we have to make sure that we use specific version across the board. fetching the latest docker image or the latest prefect core version is not something one should do in production, unless one makes sure to also update his Prefect Server (which might not be possible or required in production environments. attached is the latest error we received out of the blue, but this is not the first time we experience a breaking change.
Unexpected error: ClientError('400 Client Error: Bad Request for url: <>\n\nThis is likely caused by a poorly formatted GraphQL query or mutation. GraphQL sent:\n\nquery {\n    mutation($input: create_task_run_artifact_input!) {\n            create_task_run_artifact(input: $input) {\n                id\n        }\n    }\n}\nvariables {\n    {"input": {"task_run_id": "5b4f7865-b192-462e-a9ee-753e243db30a", "kind": "link", "data": {"link": "/default/flow-run/4fc7c743-787f-49f7-b3dc-31cfc2fa720f"}, "tenant_id": null}}\n}')
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/prefect/engine/", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/engine/", line 856, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/", line 298, in run_task_with_timeout
    return*args, **kwargs)  # type: ignore
  File "/usr/local/lib/python3.8/site-packages/prefect/utilities/", line 449, in method
    return run_method(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/prefect/tasks/prefect/", line 173, in run
  File "/usr/local/lib/python3.8/site-packages/prefect/", line 31, in create_link
    return client.create_task_run_artifact(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/", line 1689, in create_task_run_artifact
    result = self.graphql(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/", line 296, in graphql
    result = <|>(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/", line 211, in post
    response = self._request(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/", line 433, in _request
    response = self._send_request(
  File "/usr/local/lib/python3.8/site-packages/prefect/client/", line 365, in _send_request
    raise ClientError(msg)
prefect.utilities.exceptions.ClientError: 400 Client Error: Bad Request for url: <>

This is likely caused by a poorly formatted GraphQL query or mutation. GraphQL sent:

query {
    mutation($input: create_task_run_artifact_input!) {
            create_task_run_artifact(input: $input) {
variables {
    {"input": {"task_run_id": "5b4f7865-b192-462e-a9ee-753e243db30a", "kind": "link", "data": {"link": "/default/flow-run/4fc7c743-787f-49f7-b3dc-31cfc2fa720f"}, "tenant_id": null}}

Jim Crist-Harif

11/25/2020, 4:53 PM
Hi Roey, Apologies for the rough experience you're having here. Our current policy for
deployments is that the version of
should always be >= the version of
deployed (
is backwards-compatible with
, but not the other way around). Prefect is a rapidly developed library, with new apis being added to the backend to support new features all the time. We're unlikely to add a more rigorous process around mutual-backwards-compatibility as it would lead to a slower development cycle for us. The only "officially supported" version of
is a local deployment using
prefect server start
(which implies that all versions match). Deployments outside this scope are unsupported (though we'll still try to answer questions and resolve bugs), and assume that the user takes on the burden of handling smooth upgrades. In summary: -
prefect-server >= prefect
should always work -
prefect-server < prefect
might work, but isn't guaranteed and is unsupported.
🙌 1
With regards to "semver" around the
library - the major/minor/patch indicators there have to do with user-facing python api changes, not changes with backend compatibility. Since the 0.13.17 release didn't break any user-facing apis, it's still a patch release (even though it's not compatible with all previous releases of
, which as stated above should always be >= the version of

Roey Brecher

11/26/2020, 9:31 AM
thanks for the clarification.