Hi All, Some of our prefect flows get stuck in Pending state and never run after that (happens intermittently). They run on EKS pods. I see this error a lot in prefect agent logs (pasted in thread), is there any bug with prefect? Using version 2.10.13

Hi everyone, I try to disable the telemetry service in Prefect by using the config.toml and currently trying to verify if the telemetry is already disable or not. I try to output the logs in .prefect folder but it seems like no logs appeared in .prefect folder. May I know if you have any idea on how to verify it?

Hey everyone! I managed to write the necessary Infrastructure as Code (via Pulumi) for a light-weight Prefect agent that runs as an ECS Service. I was hoping to get a nice welcome message in the logs. Unfortunately, I get the

prefect

cli help message over and over again. ECS console shows a list of

STOPPED

tasks, which I'm assuming are various attempts to start the service and keep it running.

@Marvin I create deployment, but it's not showing up in the Prefect UI

@Marvin Can you explain this error when trying to run concurrent deployments in parallel? 154459.516 | DEBUG | prefect.client - Encountered retryable exception during request. Another attempt will be made in 1.762195377774828s. This is attempt 1/6. Traceback (most recent call last): File "C:\Users\cleung\AppData\Roaming\Python\Python38\site-packages\httpx\_transports\default.py", line 60, in map_httpcore_exceptions yield File "C:\Users\cleung\AppData\Roaming\Python\Python38\site-packages\httpx\_transports\default.py", line 353, in handle_async_request resp = await self._pool.handle_async_request(req) File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_async\connection_pool.py", line 253, in handle_async_request raise exc File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_async\connection_pool.py", line 237, in handle_async_request response = await connection.handle_async_request(request) File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_async\connection.py", line 90, in handle_async_request return await self._connection.handle_async_request(request) File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_async\http11.py", line 112, in handle_async_request raise exc File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_async\http11.py", line 91, in handle_async_request ) = await self._receive_response_headers(**kwargs) File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_async\http11.py", line 155, in _receive_response_headers event = await self._receive_event(timeout=timeout) File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_async\http11.py", line 205, in _receive_event raise RemoteProtocolError(msg) httpcore.RemoteProtocolError: Server disconnected without sending a response. The above exception was the direct cause of the following exception: Traceback (most recent call last): File "C:\Users\cleung\AppData\Roaming\Python\Python38\site-packages\prefect\client\base.py", line 216, in _send_with_retry response = await request() File "C:\Users\cleung\AppData\Roaming\Python\Python38\site-packages\httpx\_client.py", line 1620, in send response = await self._send_handling_auth( File "C:\Users\cleung\AppData\Roaming\Python\Python38\site-packages\httpx\_client.py", line 1648, in _send_handling_auth response = await self._send_handling_redirects( File "C:\Users\cleung\AppData\Roaming\Python\Python38\site-packages\httpx\_client.py", line 1685, in _send_handling_redirects response = await self._send_single_request(request) File "C:\Users\cleung\AppData\Roaming\Python\Python38\site-packages\httpx\_client.py", line 1722, in _send_single_request response = await transport.handle_async_request(request) File "C:\Users\cleung\AppData\Roaming\Python\Python38\site-packages\httpx\_transports\default.py", line 353, in handle_async_request resp = await self._pool.handle_async_request(req) File "\\v-tm-qeq-05\Prefect_Env\lib\contextlib.py", line 131, in exit self.gen.throw(type, value, traceback) File "C:\Users\cleung\AppData\Roaming\Python\Python38\site-packages\httpx\_transports\default.py", line 77, in map_httpcore_exceptions raise mapped_exc(message) from exc httpx.RemoteProtocolError: Server disconnected without sending a response. 154515.504 | DEBUG | prefect.client - Encountered retryable exception during request. Another attempt will be made in 1.956168415170509s. This is attempt 1/6. Traceback (most recent call last): File "\\v-tm-qeq-05\Prefect_Env\lib\asyncio\windows_events.py", line 457, in finish_recv return ov.getresult() OSError: [WinError 64] The specified network name is no longer available During handling of the above exception, another exception occurred: Traceback (most recent call last): File "\\v-tm-qeq-05\Prefect_Env\lib\asyncio\proactor_events.py", line 280, in _loop_reading data = fut.result() File "\\v-tm-qeq-05\Prefect_Env\lib\asyncio\windows_events.py", line 812, in _poll value = callback(transferred, key, ov) File "\\v-tm-qeq-05\Prefect_Env\lib\asyncio\windows_events.py", line 461, in finish_recv raise ConnectionResetError(*exc.args) ConnectionResetError: [WinError 64] The specified network name is no longer available The above exception was the direct cause of the following exception: Traceback (most recent call last): File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_exceptions.py", line 10, in map_exceptions yield File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\backends\asyncio.py", line 34, in read return await self._stream.receive(max_bytes=max_bytes) File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\anyio\_backends\_asyncio.py", line 1212, in receive raise self._protocol.exception anyio.BrokenResourceError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\Users\cleung\AppData\Roaming\Python\Python38\site-packages\httpx\_transports\default.py", line 60, in map_httpcore_exceptions yield File "C:\Users\cleung\AppData\Roaming\Python\Python38\site-packages\httpx\_transports\default.py", line 353, in handle_async_request resp = await self._pool.handle_async_request(req) File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_async\connection_pool.py", line 253, in handle_async_request raise exc File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_async\connection_pool.py", line 237, in handle_async_request response = await connection.handle_async_request(request) File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_async\connection.py", line 90, in handle_async_request return await self._connection.handle_async_request(request) File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_async\http11.py", line 112, in handle_async_request raise exc File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_async\http11.py", line 91, in handle_async_request ) = await self._receive_response_headers(**kwargs) File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_async\http11.py", line 155, in _receive_response_headers event = await self._receive_event(timeout=timeout) File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_async\http11.py", line 191, in _receive_event data = await self._network_stream.read( File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\backends\asyncio.py", line 36, in read return b"" File "\\v-tm-qeq-05\Prefect_Env\lib\contextlib.py", line 131, in exit self.gen.throw(type, value, traceback) File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_exceptions.py", line 14, in map_exceptions raise to_exc(exc) httpcore.ReadError The above exception was the direct cause of the following exception: Traceback (most recent call last): File "C:\Users\cleung\AppData\Roaming\Python\Python38\site-packages\prefect\client\base.py", line 216, in _send_with_retry response = await request() File "C:\Users\cleung\AppData\Roaming\Python\Python38\site-packages\httpx\_client.py", line 1620, in send response = await self._send_handling_auth( File "C:\Users\cleung\AppData\Roaming\Python\Python38\site-packages\httpx\_client.py", line 1648, in _send_handling_auth response = await self._send_handling_redirects( File "C:\Users\cleung\AppData\Roaming\Python\Python38\site-packages\httpx\_client.py", line 1685, in _send_handling_redirects response = await self._send_single_request(request) File "C:\Users\cleung\AppData\Roaming\Python\Python38\site-packages\httpx\_client.py", line 1722, in _send_single_request response = await transport.handle_async_request(request) File "C:\Users\cleung\AppData\Roaming\Python\Python38\site-packages\httpx\_transports\default.py", line 353, in handle_async_request resp = await self._pool.handle_async_request(req) File "\\v-tm-qeq-05\Prefect_Env\lib\contextlib.py", line 131, in exit self.gen.throw(type, value, traceback) File "C:\Users\cleung\AppData\Roaming\Python\Python38\site-packages\httpx\_transports\default.py", line 77, in map_httpcore_exceptions raise mapped_exc(message) from exc httpx.ReadError 154515.690 | DEBUG | prefect.client - Encountered retryable exception during request. Another attempt will be made in 1.9267586944460113s. This is attempt 1/6. Traceback (most recent call last): File "\\v-tm-qeq-05\Prefect_Env\lib\asyncio\windows_events.py", line 457, in finish_recv return ov.getresult() OSError: [WinError 64] The specified network name is no longer available During handling of the above exception, another exception occurred: Traceback (most recent call last): File "\\v-tm-qeq-05\Prefect_Env\lib\asyncio\proactor_events.py", line 280, in _loop_reading data = fut.result() File "\\v-tm-qeq-05\Prefect_Env\lib\asyncio\windows_events.py", line 812, in _poll value = callback(transferred, key, ov) File "\\v-tm-qeq-05\Prefect_Env\lib\asyncio\windows_events.py", line 461, in finish_recv raise ConnectionResetError(*exc.args) ConnectionResetError: [WinError 64] The specified network name is no longer available The above exception was the direct cause of the following exception: Traceback (most recent call last): File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_exceptions.py", line 10, in map_exceptions yield File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\backends\asyncio.py", line 34, in read return await self._stream.receive(max_bytes=max_bytes) File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\anyio\_backends\_asyncio.py", line 1212, in receive raise self._protocol.exception anyio.BrokenResourceError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "C:\Users\cleung\AppData\Roaming\Python\Python38\site-packages\httpx\_transports\default.py", line 60, in map_httpcore_exceptions yield File "C:\Users\cleung\AppData\Roaming\Python\Python38\site-packages\httpx\_transports\default.py", line 353, in handle_async_request resp = await self._pool.handle_async_request(req) File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_async\connection_pool.py", line 253, in handle_async_request raise exc File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_async\connection_pool.py", line 237, in handle_async_request response = await connection.handle_async_request(request) File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_async\connection.py", line 90, in handle_async_request return await self._connection.handle_async_request(request) File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_async\http11.py", line 112, in handle_async_request raise exc File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_async\http11.py", line 91, in handle_async_request ) = await self._receive_response_headers(**kwargs) File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_async\http11.py", line 155, in _receive_response_headers event = await self._receive_event(timeout=timeout) File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_async\http11.py", line 191, in _receive_event data = await self._network_stream.read( File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\backends\asyncio.py", line 36, in read return b"" File "\\v-tm-qeq-05\Prefect_Env\lib\contextlib.py", line 131, in exit self.gen.throw(type, value, traceback) File "\\v-tm-qeq-05\Prefect_Env\lib\site-packages\httpcore\_exceptions.py", line 14, in map_exceptions raise to_exc(exc) httpcore.ReadError

I’ve have a scheduled flow that is failing due to a credential issue:

Error downloading Flow from S3: An error occurred (ExpiredToken) when calling the GetObject operation: The provided token has expired.

I’m using AWS SSO credentials to upload the flow to S3:

flow.storage = S3(
    bucket=Config.PREFECT_S3_BUCKET,
    client_options={
        "aws_access_key_id": Config.AWS_ACCESS_KEY_ID,
        "aws_secret_access_key": Config.AWS_SECRET_ACCESS_KEY,
        "aws_session_token": Config.AWS_SESSION_TOKEN,
    },
)

Initially, the flow runs. However, flows scheduled after the token expiry fail with the above error. How do I both use SSO to upload flows to S3 and have them scheduled on Prefect Cloud?

Prefect Tasks library in 2.0

Hi! I intend to get Prefect to send alert only when a flow failed, but it’s sending email on every run, what have I done wrongly here (code attached)? If I also intend to get an alert when output is above certain value (output>2), should I use state_handler or something else? Thank you!!!

@Marvin when I use an RSA key for my Prefect Snowflake Connector block - what text to I put in the private key box? the value with or with out the BEGIN and END statements?

@Marvin: What happened to SequentialTaskRunner in Prefect 3.0?