Kevin
09/15/2023, 7:45 PMVicky Kwan
09/15/2023, 10:35 PMTraceback (most recent call last):
File "/opt/hostedtoolcache/Python/3.10.13/x64/bin/prefect", line 5, in <module>
from prefect.cli import app
File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/prefect/__init__.py", line 37, in <module>
from prefect.runner import Runner, serve
File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/prefect/runner.py", line 46, in <module>
from prefect.client.orchestration import get_client
File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/prefect/client/orchestration.py", line 17, in <module>
import prefect.settings
File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/prefect/settings.py", line 639, in <module>
PREFECT_CLOUD_URL = Setting(
File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/prefect/settings.py", line 120, in __init__
generate_deprecation_message(
File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/prefect/_internal/compatibility/deprecated.py", line 63, in generate_deprecation_message
parsed_start_date = pendulum.from_format(start_date, DEPRECATED_DATEFMT)
File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pendulum/__init__.py", line 259, in from_format
parts = _formatter.parse(string, fmt, now(), locale=locale)
File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pendulum/__init__.py", line 211, in now
dt = _datetime.datetime.now(local_timezone())
File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pendulum/tz/__init__.py", line 60, in local_timezone
return get_local_timezone()
File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pendulum/tz/local_timezone.py", line 35, in get_local_timezone
tz = _get_system_timezone()
File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pendulum/tz/local_timezone.py", line 63, in _get_system_timezone
return _get_unix_timezone()
File "/opt/hostedtoolcache/Python/3.10.13/x64/lib/python3.10/site-packages/pendulum/tz/local_timezone.py", line 242, in _get_unix_timezone
raise RuntimeError("Unable to find any timezone configuration")
RuntimeError: Unable to find any timezone configuration
Error: Process completed with exit code 1.
GoshDarnedHero
09/19/2023, 3:54 PM12:51:35.373 | INFO | prefect.runner - Pausing schedules for all deployments...
12:52:40.797 | INFO | prefect.runner - All deployment schedules have been paused!
Traceback (most recent call last):
File "/app/scripts/jobs.py", line 140, in <module>
Failed the last 3 attempts. Please check your environment and configuration.
Examples of recent errors:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/h2/connection.py", line 224, in process_input
func, target_state = self._transitions[(self.state, input_)]
KeyError: (<ConnectionState.CLOSED: 3>, <ConnectionInputs.SEND_HEADERS: 0>)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/httpcore/_async/http2.py", line 106, in handle_async_request
await self._send_request_headers(request=request, stream_id=stream_id)
File "/usr/local/lib/python3.10/dist-packages/httpcore/_async/http2.py", line 203, in _send_request_headers
self._h2_state.send_headers(stream_id, headers, end_stream=end_stream)
File "/usr/local/lib/python3.10/dist-packages/h2/connection.py", line 766, in send_headers
self.state_machine.process_input(ConnectionInputs.SEND_HEADERS)
File "/usr/local/lib/python3.10/dist-packages/h2/connection.py", line 228, in process_input
raise ProtocolError(
h2.exceptions.ProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
d
File "/usr/local/lib/python3.10/dist-packages/httpx/_transports/default.py", line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.10/dist-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
raise exc
File "/usr/local/lib/python3.10/dist-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
response = await connection.handle_async_request(request)
serve(
File "/usr/local/lib/python3.10/dist-packages/prefect/utilities/asyncutils.py", line 255, in coroutine_wrapper
File "/usr/local/lib/python3.10/dist-packages/httpcore/_async/connection.py", line 90, in handle_async_request
return await self._connection.handle_async_request(request)
File "/usr/local/lib/python3.10/dist-packages/httpcore/_async/http2.py", line 142, in handle_async_request
raise LocalProtocolError(exc) # pragma: nocover
httpcore.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/dist-packages/prefect/utilities/services.py", line 53, in critical_service_loop
await workload()
File "/usr/local/lib/python3.10/dist-packages/prefect/runner.py", line 501, in _get_and_submit_flow_runs
runs_response = await self._get_scheduled_flow_runs()
File "/usr/local/lib/python3.10/dist-packages/prefect/runner.py", line 614, in _get_scheduled_flow_runs
scheduled_flow_runs = await self._client.read_flow_runs(
File "/usr/local/lib/python3.10/dist-packages/prefect/client/orchestration.py", line 1778, in read_flow_runs
response = await <http://self._client.post|self._client.post>("/flow_runs/filter", json=body)
File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 1848, in post
return await self.request(
File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 1533, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/usr/local/lib/python3.10/dist-packages/prefect/client/base.py", line 256, in send
response = await self._send_with_retry(
File "/usr/local/lib/python3.10/dist-packages/prefect/client/base.py", line 190, in _send_with_retry
response = await request()
File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 1620, in send
response = await self._send_handling_auth(
return call()
File "/usr/local/lib/python3.10/dist-packages/prefect/_internal/concurrency/calls.py", line 382, in __call__
File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 1648, in _send_handling_auth
response = await self._send_handling_redirects(
File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 1685, in _send_handling_redirects
response = await self._send_single_request(request)
File "/usr/local/lib/python3.10/dist-packages/httpx/_client.py", line 1722, in _send_single_request
response = await transport.handle_async_request(request)
File "/usr/local/lib/python3.10/dist-packages/httpx/_transports/default.py", line 352, in handle_async_request
with map_httpcore_exceptions():
File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.10/dist-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.LocalProtocolError: Invalid input ConnectionInputs.SEND_HEADERS in state ConnectionState.CLOSED
return self.result()
File "/usr/local/lib/python3.10/dist-packages/prefect/_internal/concurrency/calls.py", line 282, in result
return self.future.result(timeout=timeout)
File "/usr/local/lib/python3.10/dist-packages/prefect/_internal/concurrency/calls.py", line 168, in result
return self.__get_result()
File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
raise self._exception
File "/usr/local/lib/python3.10/dist-packages/prefect/_internal/concurrency/calls.py", line 345, in _run_async
result = await coro
File "/usr/local/lib/python3.10/dist-packages/prefect/runner.py", line 989, in serve
await runner.start()
File "/usr/local/lib/python3.10/dist-packages/prefect/runner.py", line 283, in start
async with self._loops_task_group as tg:
File "/usr/local/lib/python3.10/dist-packages/anyio/_backends/_asyncio.py", line 597, in __aexit__
raise exceptions[0]
File "/usr/local/lib/python3.10/dist-packages/prefect/utilities/services.py", line 121, in critical_service_loop
raise RuntimeError("Service exceeded error threshold.")
RuntimeError: Service exceeded error threshold.
GoshDarnedHero
09/20/2023, 2:30 AM/usr/lib/python3.10/runpy.py:126: RuntimeWarning: 'prefect.engine' found in sys.modules after import of package 'prefect', but prior to execution of 'prefect.engine'; this may result in unpredictable behaviour
warn(RuntimeWarning(msg))
GoshDarnedHero
09/20/2023, 2:30 AMSergey Zakharchenko
09/20/2023, 8:49 AM.submit()
, and final task with wait_for
for all previous tasks. I also have decorator @task
with setting retries=0
. In addition, I explicitly switched global Prefect settings like this:
prefect config set PREFECT_FLOW_DEFAULT_RETRIES=0
prefect config set PREFECT_TASK_DEFAULT_RETRIES=0
Nevertheless, I still can see retrying
and rerunning
states of the tasks. It's happening without any messages in logs. Some tasks cannot actually finish because of "repeating". But I want tasks to be executed only once. Did I do something wrong?
Some pictures will be below ⤵️Johan sh
09/20/2023, 2:57 PMprefect deploy
:
prefect.exceptions.ScriptError: Script at 'xxxxxxxx.py' encountered an exception: PrefectHTTPStatusError("Client error '401 Unauthorized' for url '<https://api.prefect.cloud/api/accounts/xxxxxxx/workspaces/xxxxxx/block_types/slug/gcp-credentials/block_documents/name/bq-load-creds?include_secrets=true>'\nResponse: {'detail': 'Invalid authentication credentials'}\nFor more information check: <https://httpstatuses.com/401>"
It seems that there is an issue with the gcp-credential block. I also have the same issue with other types of blocks and on flows that I successfully deployed in the past... (I'm using Prefect Cloud)
Can somebody help me please 😞?Brian Newman
09/20/2023, 9:11 PMJohn Lemmon
09/21/2023, 2:41 PMSantiago Gonzalez
09/21/2023, 9:45 PMTraceback (most recent call last):
File "/home/sgonzalez/venv/lib/python3.8/site-packages/prefect/engine.py", line 1729, in orchestrate_task_run
result = await call.aresult()
File "/home/sgonzalez/venv/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 291, in aresult
return await asyncio.wrap_future(self.future)
File "/home/sgonzalez/venv/lib/python3.8/site-packages/prefect/_internal/concurrency/calls.py", line 345, in _run_async
result = await coro
File "/home/sgonzalez/venv/lib/python3.8/site-packages/prefect_databricks/jobs.py", line 1170, in jobs_runs_get
response = await execute_endpoint.fn(
File "/home/sgonzalez/venv/lib/python3.8/site-packages/prefect_databricks/rest.py", line 134, in execute_endpoint
response = await getattr(client, http_method)(
File "/home/sgonzalez/venv/lib/python3.8/site-packages/httpx/_client.py", line 1757, in get
return await self.request(
File "/home/sgonzalez/venv/lib/python3.8/site-packages/httpx/_client.py", line 1530, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/home/sgonzalez/venv/lib/python3.8/site-packages/httpx/_client.py", line 1617, in send
response = await self._send_handling_auth(
File "/home/sgonzalez/venv/lib/python3.8/site-packages/httpx/_client.py", line 1645, in _send_handling_auth
response = await self._send_handling_redirects(
File "/home/sgonzalez/venv/lib/python3.8/site-packages/httpx/_client.py", line 1682, in _send_handling_redirects
response = await self._send_single_request(request)
File "/home/sgonzalez/venv/lib/python3.8/site-packages/httpx/_client.py", line 1719, in _send_single_request
response = await transport.handle_async_request(request)
File "/home/sgonzalez/venv/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/home/sgonzalez/venv/lib/python3.8/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectTimeout
07:34:11 PM
jobs_runs_get-173
prefect.task_runs
Finished in state Failed('Task run encountered an exception ConnectTimeout: ')
The databricks job still runs, but the flow had already finished.Louis Vines
09/22/2023, 1:07 PMBen Hairston
09/22/2023, 4:06 PMEric
09/22/2023, 8:53 PMJonah
09/22/2023, 11:40 PMBen Epstein
09/24/2023, 5:52 PMprefect cloud login
When I try to auth with the browser I see the following screenshot. But when I try to use my key from https://app.prefect.cloud/my/api-keys
I get the error
Unable to authenticate with Prefect Cloud. Your key is not in our expected format: 'pnu_' or 'pnb_'.
Can you provide some tips?Ben Epstein
09/24/2023, 8:42 PMWorkers are lightweight polling services that retrieve scheduled runs from a work pool and execute them.
Agent processes are lightweight polling services that get scheduled work from a work pool and deploy the corresponding flow runs.I have a work-pool defined as
process
, and i see the same behavior via
prefect agent start --pool default-process-1
and
prefect worker start --pool default-process-1
I'm similarly confused by work-pool types local
and prefect-agent
-- Is the difference simply that a prefect-agent
work-pool can only be picked up by agents, whereas a local
workpool can picked up by either?
Thanks!Sagar Dhande
09/25/2023, 2:19 AMCasey M
09/25/2023, 7:00 PMHugo Vandernotte
09/26/2023, 9:43 AMSoami Charan
09/27/2023, 7:36 AMJohn Mizerany
09/27/2023, 3:02 PMKevin
09/27/2023, 7:30 PMGabe Villasana
09/27/2023, 9:39 PMprefect.exceptions.PrefectHTTPStatusError: Server error '503 Service Unavailable' for url
Joseph Thickpenny Ryan
09/28/2023, 8:49 AMPending
state? We have a flow that runs multiple instances of a deployment but is struggling to initialise more than 10 instances simultaneouslyBrian Newman
09/29/2023, 1:09 AMStephen Herron
09/29/2023, 10:28 AMSubflow Runs
tabIvan
10/02/2023, 1:20 PMFlow, flow run, and task run rate limits
— are those limits includes API methods to retrieve connection blocks data?
2) Log service rate limits
— each new line in logs counts?GoshDarnedHero
10/03/2023, 3:08 AMValueError: the greenlet library is required to use this function. No module named 'greenlet'
GoshDarnedHero
10/03/2023, 3:08 AMGoshDarnedHero
10/03/2023, 3:09 AM