Hi, we just had a two flows becoming stuck without any apparent reason. All tasks had been finished but the flow would remain as running, therefore blocking others. Any idea how to debug what was going on? Manually setting the state resolved the problem.

Hey everyone, I'm getting a weird error when trying to run flows with the docker agent against the cloud backend. The error message asks me to make sure I'm authenticated. However, when I run

prefect auth list-tenants

I get my tenants, so if feel like I should be authenticated. I'm kind of stumped, maybe someone here has an idea?

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 738, in _request
    json_resp = response.json()
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 900, in json
    return complexjson.loads(self.text, **kwargs)
  File "/usr/local/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/usr/local/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/bin/prefect", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.7/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/prefect/cli/execute.py", line 53, in flow_run
    result = client.graphql(query)
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 550, in graphql
    retry_on_api_error=retry_on_api_error,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 454, in post
    retry_on_api_error=retry_on_api_error,
  File "/usr/local/lib/python3.7/site-packages/prefect/client/client.py", line 744, in _request
    ) from exc
prefect.exceptions.AuthorizationError: Malformed response received from Cloud - please ensure that you are authenticated. See `prefect auth login --help`.

@Marvin I’m trying to deploy a Prefect Server locally that will run on Prefect 2.20.4 The server is currently running along with a Postgres DB at its backend. I also deployed a Prefect Worker of type Kubernetes, also using an image with Prefect 2.20.4 installed. Now, I’ve deployed a flow that uses a custom image that also uses Prefect 2.20.4. I’m trying to run the flow using the UI and am getting the following errors:

2025-01-28 16:15:28 14:15:28.087 | INFO    | prefect.flow_runs.runner - Opening process...
2025-01-28 16:15:34 /usr/local/lib/python3.8/runpy.py:127: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
2025-01-28 16:15:34   warn(RuntimeWarning(msg))
2025-01-28 16:15:34 /usr/local/lib/python3.8/site-packages/prefect/utilities/dispatch.py:162: UserWarning: Type 'KubernetesClusterConfig' at /usr/local/lib/python3.8/site-packages/prefect_kubernetes/credentials.py:40 has key 'kubernetes-cluster-config' that matches existing registered type 'KubernetesClusterConfig' from /usr/local/lib/python3.8/site-packages/prefect/blocks/kubernetes.py:32. The existing type will be overridden.
2025-01-28 16:15:34   warnings.warn(
2025-01-28 16:15:34 /usr/local/lib/python3.8/site-packages/pydantic/v1/schema.py:631: UserWarning: Callable print_func was excluded from schema since JSON schema has no equivalent type.
2025-01-28 16:15:34   warnings.warn(skip.message, UserWarning)
2025-01-28 16:15:34 /usr/local/lib/python3.8/site-packages/prefect/engine.py:436: PrefectDeprecationWarning: prefect.deployments.deployments.load_flow_from_flow_run has been deprecated. It will not be available after Dec 2024. Will be moved in Prefect 3 to prefect.flows:load_flow_from_flow_run
2025-01-28 16:15:34   else await load_flow_from_flow_run(
2025-01-28 16:15:34 14:15:34.076 | INFO    | prefect.deployment - Cloned repository '<https://github.com/*/*.git>' into '*-test_filter_flow_locally'
2025-01-28 16:15:34 14:15:34.127 | ERROR   | prefect.flow_runs.runner - Process for flow run 'artichoke-yak' exited with status code: -4
2025-01-28 16:15:34 14:15:34.152 | INFO    | prefect.flow_runs.runner - Reported flow run 'dfacf593-fa9f-496b-ac4c-efcbc41fd592' as crashed: Flow run process exited with non-zero status code -4.
2025-01-28 16:15:35 14:15:35.434 | INFO    | prefect.deployment - Cloned repository '<https://github.com/*/*.git>' into '*-test_filter_flow_locally'

I appear to be up and running, but noticed the flow run does not appear on the Prefect web console. I am authenticated and was under the impression that even if I execute a flow locally, the logs/results of the flow would go to prefect. Is that not the case with Controlflow? I apologize if this is a basic question, but still trying to wrap my head around Prefect + Controlflow.

@Marvin how do I get the return value of a deployment from the run_deployment function?

hello, i am running a self hosted prefect server and a prefect worker (k8). I am running a flow where it requires a different image, and it is deployed specifically in the flow deployment. There is random/transient error that says module not found (attached stacktrace in 🧵) when the job is trying to start, but is often fixed with a 2nd or 3rd run retry max(where i will have to trigger it manually). Sometimes of 50% chance it will succeed without any retries. I have specified retries on my flows, however whenever this error appears, the flow is never retried. The error appears to me that the pod for the flow is not properly provisioned with the right package for the flow run to happen. Any idea how do i troubleshoot or proceed further, the failure rate of this is almost 10/21 of the past 3 weeks(without manual retry).

@Marvin How to install prefect on a local server using a Docker Compose file?

@Marvin is there a native way in prefect to watch a folder on the file system and if files show up in it then trigger a flow?

Hey all! Having trouble deploying a flow on the

build_docker_image

step. traceback:

Traceback (most recent call last):
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/docker/transport/unixconn.py", line 27, in connect
    sock.connect(self.unix_socket)
FileNotFoundError: [Errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/urllib3/connectionpool.py", line 799, in urlopen
    retries = retries.increment(
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/urllib3/util/retry.py", line 550, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/urllib3/packages/six.py", line 769, in reraise
    raise value.with_traceback(tb)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/urllib3/connectionpool.py", line 715, in urlopen
    httplib_response = self._make_request(
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/urllib3/connectionpool.py", line 416, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/urllib3/connection.py", line 244, in request
    super(HTTPConnection, self).request(method, url, body=body, headers=headers)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/lib/python3.10/http/client.py", line 1283, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/lib/python3.10/http/client.py", line 1329, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/lib/python3.10/http/client.py", line 1278, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/lib/python3.10/http/client.py", line 1038, in _send_output
    self.send(msg)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/lib/python3.10/http/client.py", line 976, in send
    self.connect()
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/docker/transport/unixconn.py", line 27, in connect
    sock.connect(self.unix_socket)
urllib3.exceptions.ProtocolError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/docker/api/client.py", line 214, in _retrieve_server_version
    return self.version(api_version=False)["ApiVersion"]
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/docker/api/daemon.py", line 181, in version
    return self._result(self._get(url), json=True)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/docker/api/client.py", line 237, in _get
    return self.get(url, **self._set_request_timeout(kwargs))
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/requests/sessions.py", line 602, in get
    return self.request("GET", url, **kwargs)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/requests/adapters.py", line 501, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/prefect/utilities/dockerutils.py", line 110, in docker_client
    client = docker.DockerClient.from_env()
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/docker/client.py", line 96, in from_env
    return cls(
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/docker/client.py", line 45, in __init__
    self.api = APIClient(*args, **kwargs)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/docker/api/client.py", line 197, in __init__
    self._version = self._retrieve_server_version()
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/docker/api/client.py", line 221, in _retrieve_server_version
    raise DockerException(
docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory'))

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 154, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 125, in run_step
    result = await from_async.call_soon_in_new_thread(
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 326, in aresult
    return await asyncio.wrap_future(self.future)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 351, in _run_sync
    result = self.fn(*self.args, **self.kwargs)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/prefect/_internal/compatibility/deprecated.py", line 183, in wrapper
    return fn(*args, **kwargs)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/prefect_docker/deployments/steps.py", line 200, in build_docker_image
    with docker_client() as client:
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/lib/python3.10/contextlib.py", line 135, in __enter__
    return next(self.gen)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/prefect/utilities/dockerutils.py", line 114, in docker_client
    raise RuntimeError(
RuntimeError: This error is often thrown because Docker is not running. Please ensure Docker is running.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/prefect/utilities/asyncutils.py", line 259, in coroutine_wrapper
    return call()
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 431, in __call__
    return self.result()
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 317, in result
    return self.future.result(timeout=timeout)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 178, in result
    return self.__get_result()
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/prefect/_internal/concurrency/calls.py", line 388, in _run_async
    result = await coro
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/prefect/cli/deploy.py", line 306, in deploy
    await _run_single_deploy(
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/prefect/client/utilities.py", line 78, in with_injected_client
    return await fn(*args, **kwargs)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/prefect/cli/deploy.py", line 595, in _run_single_deploy
    await run_steps(build_steps, step_outputs, print_function=app.console.print)
  File "/Users/alex.ilyin/.pyenv/versions/3.10.12/envs/dwh/lib/python3.10/site-packages/prefect/deployments/steps/core.py", line 182, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect_docker.deployments.steps.build_docker_image

I’ve made sure that docker is running before running

prefect deploy

. any advice here?

Can anyone explain this error message:

Cannot create a project-style deployment with work pool of type 'prefect-agent'.

? Why not? Is there no way to do deployments with

prefect-agent

with a

prefect.yaml

file?