Hello all! I am attempting to have a worker deployed on a kubernetes cluster run a few flows for us, but am having some trouble getting it to clone a github repository(followed the guidance here: https://docs.prefect.io/latest/guides/deployment/storage-guide/). Not sure if there is anything else I should do here or if there are any additions needed to the script maybe?

Hi friends. I'm using Prefect2 cloud and trying to follow Laura's video on getting a hello world deployment via GH block to a k8s worker pool. I can create a new deployment in my local env and run a flow, but whenever I move to a GHA workflow (I even checked pip freeze and all look the same), I see this error:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.13/x64/bin/prefect", line 5, in <module>
    from prefect.cli import app
  File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/prefect/__init__.py", line 37, in <module>
    from prefect.runner import Runner, serve
  File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/prefect/runner.py", line 46, in <module>
    from prefect.client.orchestration import get_client
  File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/prefect/client/orchestration.py", line 17, in <module>
    import prefect.settings
  File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/prefect/settings.py", line 639, in <module>
    PREFECT_CLOUD_URL = Setting(
  File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/prefect/settings.py", line 120, in __init__
    generate_deprecation_message(
  File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/prefect/_internal/compatibility/deprecated.py", line 63, in generate_deprecation_message
    parsed_start_date = pendulum.from_format(start_date, DEPRECATED_DATEFMT)
  File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pendulum/__init__.py", line 259, in from_format
    parts = _formatter.parse(string, fmt, now(), locale=locale)
  File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pendulum/__init__.py", line 211, in now
    dt = _datetime.datetime.now(local_timezone())
  File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pendulum/tz/__init__.py", line 60, in local_timezone
    return get_local_timezone()
  File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pendulum/tz/local_timezone.py", line 35, in get_local_timezone
    tz = _get_system_timezone()
  File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pendulum/tz/local_timezone.py", line 63, in _get_system_timezone
    return _get_unix_timezone()
  File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pendulum/tz/local_timezone.py", line 242, in _get_unix_timezone
    raise RuntimeError("Unable to find any timezone configuration")
RuntimeError: Unable to find any timezone configuration
Error: Process completed with exit code 1.

I know this stack is huge, but I'm completely at a loss why my agent is crashing .. here is the traceback:

12:51:35.373 | INFO    | prefect.runner - Pausing schedules for all deployments...
12:52:40.797 | INFO    | prefect.runner - All deployment schedules have been paused!
Traceback (most recent call last):
  File "/app/scripts/jobs.py", line 140, in <module>
Failed the last 3 attempts. Please check your environment and configuration.
Examples of recent errors:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/h2/connection.py", line 224, in process_input
    func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/httpcore/_async/http2.py", line 106, in handle_async_request
    await self._send_request_headers(request=request, stream_id=stream_id)
  File "/usr/local/lib/python3.10/dist-packages/httpcore/_async/http2.py", line 203, in _send_request_headers
    self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
  File "/usr/local/lib/python3.10/dist-packages/h2/connection.py", line 766, in send_headers
    self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
  File "/usr/local/lib/python3.10/dist-packages/h2/connection.py", line 228, in process_input
    raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions

d
  File "/usr/local/lib/python3.10/dist-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/local/lib/python3.10/dist-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
    raise exc
  File "/usr/local/lib/python3.10/dist-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
    response = await connection.handle_async_request(request)
    serve(
  File "/usr/local/lib/python3.10/dist-packages/prefect/utilities/asyncutils.py", line 255, in coroutine_wrapper
  File "/usr/local/lib/python3.10/dist-packages/httpcore/_async/connection.py", line 90, in handle_async_request
    return await self._connection.handle_async_request(request)
  File "/usr/local/lib/python3.10/dist-packages/httpcore/_async/http2.py", line 142, in handle_async_request
    raise LocalProtocolError(exc)  # pragma: nocover
httpcore.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/prefect/utilities/services.py", line 53, in critical_service_loop
    await workload()
  File "/usr/local/lib/python3.10/dist-packages/prefect/runner.py", line 501, in _get_and_submit_flow_runs
    runs_response = await self._get_scheduled_flow_runs()
  File "/usr/local/lib/python3.10/dist-packages/prefect/runner.py", line 614, in _get_scheduled_flow_runs
    scheduled_flow_runs = await self._client.read_flow_runs(
  File "/usr/local/lib/python3.10/dist-packages/prefect/client/orchestration.py", line 1778, in read_flow_runs
    response = await <http://self._client.post|self._client.post>("/flow_runs/filter", json=body)
  File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 1848, in post
    return await self.request(
  File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 1533, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.10/dist-packages/prefect/client/base.py", line 256, in send
    response = await self._send_with_retry(
  File "/usr/local/lib/python3.10/dist-packages/prefect/client/base.py", line 190, in _send_with_retry
    response = await request()
  File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 1620, in send
    response = await self._send_handling_auth(
    return call()
  File "/usr/local/lib/python3.10/dist-packages/prefect/_internal/concurrency/calls.py", line 382, in __call__
  File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 1648, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 1685, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 1722, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/usr/local/lib/python3.10/dist-packages/httpx/_transports/default.py", line 352, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.10/dist-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
    return self.result()
  File "/usr/local/lib/python3.10/dist-packages/prefect/_internal/concurrency/calls.py", line 282, in result
    return self.future.result(timeout=timeout)
  File "/usr/local/lib/python3.10/dist-packages/prefect/_internal/concurrency/calls.py", line 168, in result
    return self.__get_result()
  File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
    raise self._exception
  File "/usr/local/lib/python3.10/dist-packages/prefect/_internal/concurrency/calls.py", line 345, in _run_async
    result = await coro
  File "/usr/local/lib/python3.10/dist-packages/prefect/runner.py", line 989, in serve
    await runner.start()
  File "/usr/local/lib/python3.10/dist-packages/prefect/runner.py", line 283, in start
    async with self._loops_task_group as tg:
  File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 597, in __aexit__
    raise exceptions[0]
  File "/usr/local/lib/python3.10/dist-packages/prefect/utilities/services.py", line 121, in critical_service_loop
    raise RuntimeError("Service exceeded error threshold.")
RuntimeError: Service exceeded error threshold.

Is there any way to suppress the warning: `

/usr/lib/python3.10/runpy.py:126: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
  warn(RuntimeWarning(msg))

it's driving me nuts in my log files

Hello everyone! 👋 Has someone experienced an endless retries and rerunning of tasks? I have a flow with 10 tasks, which run simultaneously with

.submit()

, and final task with

wait_for

for all previous tasks. I also have decorator

@task

with setting

retries=0

. In addition, I explicitly switched global Prefect settings like this:

prefect config set PREFECT_FLOW_DEFAULT_RETRIES=0
prefect config set PREFECT_TASK_DEFAULT_RETRIES=0

Nevertheless, I still can see

retrying

and

rerunning

states of the tasks. It's happening without any messages in logs. Some tasks cannot actually finish because of "repeating". But I want tasks to be executed only once. Did I do something wrong? Some pictures will be below ⤵️

Hello everyone ! I have this error when I try to deploy a flow using

prefect deploy

:

prefect.exceptions.ScriptError: Script at 'xxxxxxxx.py' encountered an exception: PrefectHTTPStatusError("Client error '401 Unauthorized' for url '<https://api.prefect.cloud/api/accounts/xxxxxxx/workspaces/xxxxxx/block_types/slug/gcp-credentials/block_documents/name/bq-load-creds?include_secrets=true>'\nResponse: {'detail': 'Invalid authentication credentials'}\nFor more information check: <https://httpstatuses.com/401>"

It seems that there is an issue with the gcp-credential block. I also have the same issue with other types of blocks and on flows that I successfully deployed in the past... (I'm using Prefect Cloud) Can somebody help me please 😞?

Quick Question! When I am running prefect deploy --all, it runs the build and push for every deployment. How can those run once and then deploy the flows to Prefect?

I was hoping to programmatically define my work pools so we don't need to update them in the UI. But it looks like the only options are the UI or the API. The CLI can create work pools but not edit their configuration, if I'm reading this right. Is it more typical to just define these via the UI instead?

Hi , I ran a flow a couple of days ago, which runs a databricks job, and wait until it finishes. In the meanwhile it is running, the flow died within this error:

Traceback (most recent call last):
  File "/home/sgonzalez/venv/lib/python3.8/site-packages/prefect/engine.py", line 1729, in orchestrate_task_run
    result = await call.aresult()
  File "/home/sgonzalez/venv/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 291, in aresult
    return await asyncio.wrap_future(self.future)
  File "/home/sgonzalez/venv/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 345, in _run_async
    result = await coro
  File "/home/sgonzalez/venv/lib/python3.8/site-packages/prefect_databricks/jobs.py", line 1170, in jobs_runs_get
    response = await execute_endpoint.fn(
  File "/home/sgonzalez/venv/lib/python3.8/site-packages/prefect_databricks/rest.py", line 134, in execute_endpoint
    response = await getattr(client, http_method)(
  File "/home/sgonzalez/venv/lib/python3.8/site-packages/httpx/_client.py", line 1757, in get
    return await self.request(
  File "/home/sgonzalez/venv/lib/python3.8/site-packages/httpx/_client.py", line 1530, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/home/sgonzalez/venv/lib/python3.8/site-packages/httpx/_client.py", line 1617, in send
    response = await self._send_handling_auth(
  File "/home/sgonzalez/venv/lib/python3.8/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
    response = await self._send_handling_redirects(
  File "/home/sgonzalez/venv/lib/python3.8/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
    response = await self._send_single_request(request)
  File "/home/sgonzalez/venv/lib/python3.8/site-packages/httpx/_client.py", line 1719, in _send_single_request
    response = await transport.handle_async_request(request)
  File "/home/sgonzalez/venv/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
    resp = await self._pool.handle_async_request(req)
  File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/sgonzalez/venv/lib/python3.8/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectTimeout
07:34:11 PM
jobs_runs_get-173
prefect.task_runs
Finished in state Failed('Task run encountered an exception ConnectTimeout: ')

The databricks job still runs, but the flow had already finished.

I have a scenario where have a flow that queries ~ 300 apis as separate tasks. Because 1 or 2 of these tasks fail randomly and intermittently on most flow runs we have found our alerting very noisy and want to resolve this. Our solution we want is that we want to only send an alert to slack if THE SAME task fails on consecutive flow runs. Is there any way to introspect the prefect cloud state programatically (CLI or python client) to work out which tasks failed for a given flow run?

Hey quick question, is there a Prefect Cloud API Postman Collection?

What is the recommended way to update Prefect Deployments in CI/CD? For example, after I build a new image I want to update the Deployment to point to it. The docs show how to do it manually (log into prefect in terminal and then do `prefect deploy`; what credentials should we use for programatic push?

Hi, is there a way to allow runner/developer access only in Prefect Cloud UI but not via CLI?

Hello, I'm having issues authenticating to prefect with

prefect cloud login

When I try to auth with the browser I see the following screenshot. But when I try to use my key from https://app.prefect.cloud/my/api-keys I get the error

Unable to authenticate with Prefect Cloud. Your key is not in our expected format: 'pnu_' or 'pnb_'.

Can you provide some tips?

What is the difference between a worker and an agent? From the docs

Workers are lightweight polling services that retrieve scheduled runs from a work pool and execute them.

Agent processes are lightweight polling services that get scheduled work from a work pool and deploy the corresponding flow runs.

I have a work-pool defined as

process

, and i see the same behavior via

prefect agent start --pool default-process-1

and

prefect worker start --pool default-process-1

I'm similarly confused by work-pool types

local

and

prefect-agent

-- Is the difference simply that a

prefect-agent

work-pool can only be picked up by agents, whereas a

local

workpool can picked up by either? Thanks!

Hello, can someone help me clear this doubt: While using Prefect 2, if a flow fails then the retry option would run the entire flow again or it would resume the flow from the task or sub-flow where it failed? I had tried this previously and it restarted the entire flow, curious to know if this is made available in any new version. This particular feature works in Prefect 1 though.

Hi all. I just started using prefect and am trying to deploy my first flow. I can successfully deploy a python file, but the error I get is: Failed to start process for flow run 'd7c862c3-f1f2-4cbd-b00d-b1248a9add0f'. FileNotFoundError: [Errno 2] No such file or directory: '/Users/caseym/PycharmProjects/virtual' Is this because I deployed while in a virtual environment? I'm using PyCharm. Any tips?

Hey all, What are the recommandations to make Prefect access resources from different vpcs? My use case => I have several internal databases (not exposed to internet) that are running in different vpcs and I would like Prefect to query them. I am wondering what are the recommandations to do so? Or if there is a guide I could follow 🙂

Hi, I am using Prefect Cloud, I created one deployment for one work-pool which has one work queue. Now in kubernetes, I created a Process Worker pod and I scaled it to 10 pods horizontally. I put 10 as limit of flows should run on each worker. Now if I create 100 flows, so technically, all 10 prefect worker (process type) pods should pick 10 flows and execute in paralelly. But that is not happening. There is any guide how to scale workers on single work-pool to execute flows parallelly?

This message contains interactive elements.

Hello all! I am attempting to run a deployment/local and the flow seems to be saying late for an inordinate amount of time. I attempted to cancel the original flow but it seems to be showing the 'cancelling' message for over 10 minutes. Any advice here would be great, thank you!

Is the prefect cloud API down? Seeing the following:

prefect.exceptions.PrefectHTTPStatusError: Server error '503 Service Unavailable' for url

What debugging can we do to figure out why a flow would be stuck in a

Pending

state? We have a flow that runs multiple instances of a deployment but is struggling to initialise more than 10 instances simultaneously

When trying to log in I am not getting the magic link email. I have tried with a couple of accounts. Is anyone else having this issue?

not sure if this is a new bug but all flows using the subflow pattern are not listing anything under the

Subflow Runs

tab

Hi team! I have the question regarding API limits. 1)

Flow, flow run, and task run rate limits

— are those limits includes API methods to retrieve connection blocks data? 2)

Log service rate limits

— each new line in logs counts?

I'm getting the error

ValueError: the greenlet library is required to use this function. No module named 'greenlet'

when. running a flow

is that a dependancy for prefect? Because it's not installed