https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> - I've deployed prefect using prefect...
# marvin-ai
j
@Marvin - I've deployed prefect using prefect-helm, and it's been working more or less OK for about two years. I keep trimming and vacuuming prefect's postgres database automatically. Yesterday, I switched prefect to use redis (also deployed via prefect-helm). I now have this working, except the flows are not being scheduled ahead of time, so they aren't being executed. It's not clear to me where the scheduler is supposed to be running. When I run 
prefect server services ls
, no service appears representing a scheduler, enabled or disabled. There appears to be a scheduler service: https://github.com/PrefectHQ/prefect/blob/4e141b688237d1b0b4722fab1de9e43d058957e1/src/prefect/settings/models/server/services.py#L371 , https://docs.prefect.io/v3/api-ref/python/prefect-server-services-scheduler There also appears to be an environment variable that enables and disables it: 
PREFECT_API_SERVICES_SCHEDULER_ENABLED
It is enabled for me according to 
prefect config view
PREFECT_API_SERVICES_SCHEDULER_ENABLED
does not appear in prefect-helm, so it can't have been configured. Where does the scheduler run, and how can I best debug it?
m
✅ thought for 139.5 seconds
What you’re seeing is consistent with how Prefect 3.x currently works. Short version: - The scheduler does exist - It is not shown by 
prefect server services ls
- In Helm/self-hosted setups, it should run in the process/container running 
prefect server services start
(or inside 
prefect server start
if you are not using 
--no-services
) - If you moved to Redis, the most important thing to check is whether your background services process is actually running and whether 
PREFECT_SERVER_DOCKET_URL
is set correctly A few key references: - Scheduler settings source: https://github.com/PrefectHQ/prefect/blob/4e141b688237d1b0b4722fab1de9e43d058957e1/src/prefect/settings/models/server/services.py#L371 - Scheduler API docs: https://docs.prefect.io/v3/api-ref/python/prefect-server-services-scheduler - Prefect CLI help confirms 
prefect server services ls
only lists “available services and their status”, but in practice it does not include the scheduler’s perpetual-service implementation Why 
prefect server services ls
doesn’t show the scheduler In Prefect 3, the scheduler is implemented differently from the older service classes. It’s registered as a perpetual service, not one of the 
Service
subclasses that 
prefect server services ls
enumerates. So this is the important bit: - 
prefect server services ls
not showing a scheduler does not mean the scheduler is absent or disabled - The scheduler is still controlled by settings like 
PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED
Also, the env var name for Prefect 3.x is: - 
PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED
not 
PREFECT_API_SERVICES_SCHEDULER_ENABLED
If 
prefect config view
is showing the scheduler enabled, that part is probably fine. Where the scheduler runs In self-hosted server deployments, there are two common patterns: 1. Single-process/server setup - 
prefect server start
- API + background services run together 2. Production / Helm-style split setup - API container(s): 
prefect server start --no-services
- Background services container(s): 
prefect server services start
In the split setup, the scheduler should run in the background services pod/container, not the API pod. So if you’re using 
prefect-helm
, the first question is: - Do you currently have a background services deployment/pod running 
prefect server services start
? If not, that would explain exactly why no future runs are being created. What changed when you enabled Redis Redis is important for coordination in distributed/self-hosted Prefect 3 setups. In particular, the scheduler and other background services use the Docket backend for coordination/locking. The key setting is: - 
PREFECT_SERVER_DOCKET_URL
For a distributed deployment, this should point to Redis, for example:
Copy code
bash
PREFECT_SERVER_DOCKET_URL=<redis://redis:6379/1>
If this is left at the default in-memory value (
memory://
), then coordination won’t work correctly across pods/processes. That can cause confusing behavior: - duplicated service work in multi-pod setups - services not coordinating as expected - scheduler behavior that looks inconsistent What I would check first 1. Confirm which pod is supposed to run background services Look for a deployment/container command like:
Copy code
bash
prefect server services start
If your API pod is running:
Copy code
bash
prefect server start --no-services
then you must have a separate services pod. 2. Check the actual config in the background-services pod In the pod that should be running services:
Copy code
bash
prefect config view
Look for at least: - 
PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED
- 
PREFECT_SERVER_DOCKET_URL
- Redis messaging settings, if configured 3. Check whether Docket is pointed at Redis You want something like:
Copy code
bash
PREFECT_SERVER_DOCKET_URL=redis://...
If it says 
memory://
, that’s a red flag in Kubernetes / multi-pod deployments.
4. Check logs from the services pod The best debugging step is usually:
Copy code
bash
kubectl logs <background-services-pod>
or restart that pod with debug logging enabled:
Copy code
bash
PREFECT_LOGGING_LEVEL=DEBUG
Then look for messages related to: - perpetual services registration - scheduler loops - deployment scheduling - Redis/Docket connection errors 5. Verify that scheduled runs are actually being created in the DB/UI The scheduler’s job is to create future flow runs in 
Scheduled
state. If none are being created ahead of time, either: - the scheduler isn’t running - it can’t acquire coordination/lock state - the deployment schedules are inactive/misconfigured - there are scheduler errors in logs Important Helm-specific point You noted that 
PREFECT_SERVER_SERVICES_SCHEDULER_ENABLED
doesn’t appear in 
prefect-helm
. That’s not necessarily a problem. Since the default is enabled, the chart may simply not expose it explicitly unless you override it. What does matter is whether the chart is deploying: - API server(s) - background services process - correct shared PostgreSQL - correct Redis-backed Docket config So the missing env var in Helm values alone is probably not the root cause. Best way to debug this I’d suggest this sequence: 1. Find the service runner pod/deployment - confirm there is one - confirm it runs 
prefect server services start
2. Inspect config inside that pod - 
prefect config view | grep -i SCHEDULER
- 
prefect config view | grep -i DOCKET
3. Inspect logs - search for 
scheduler
- search for 
perpetual
- search for 
redis
- search for 
docket
4. Confirm deployments still have active schedules In the UI or via CLI/API, verify the deployment’s schedule is still active. 5. Check whether switching to Redis also changed service topology Sometimes Redis gets enabled, but the dedicated background-services deployment isn’t updated with the same env/config as the API pods. What I suspect most strongly Given your description, the most likely causes are: - There is no 
prefect server services start
process running anymore, or - The background services pod does not have the right Redis/Docket settings, especially 
PREFECT_SERVER_DOCKET_URL
If you want, paste: - your relevant 
prefect-helm
values for server/background services/redis - the command for the API pod and any background-services pod - output of 
prefect config view | grep -Ei 'SCHEDULER|DOCKET|REDIS|MESSAGING'
and I can help narrow it down.
j
@Marvin The following pods are running:
Copy code
NAME                                                 READY   STATUS    RESTARTS      AGE
prefect-postgresql-0                                 1/1     Running   0             16m
prefect-redis-master-0                               1/1     Running   0             16m
prefect-server-777f8976d5-bv7nk                      1/1     Running   1 (15m ago)   16m
prefect-server-background-services-cdb9d755d-9s5qx   1/1     Running   3 (15m ago)   16m
The pod 
prefect-server-background-services-cdb9d755d-9s5qx
is running 
prefect server services start
. Both pods appear to be configured correctly, to talk to redis. 
PREFECT_SERVER_DOCKET_URL
is set correctly. In the logs, there is evidence that both the 
prefect-server
and 
prefect-server-background-services
pods can access redis. Now I see a log error related to scheduling for the first time - it's a database timeout:
Copy code
16:03:36.370 | ERROR   | docket.dependencies - ↩ [ 27143ms] schedule_deployments(){schedule_deployments}
Traceback (most recent call last):
  File "/usr/local/lib/python3.13/site-packages/docket/worker.py", line 1002, in _execute
    result = await execution.function(
             ^^^^^^^^^^^^^^^^^^^^^^^^^
        *execution.args, **final_kwargs
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/prefect/server/services/scheduler.py", line 248, in schedule_deployments
    result = await session.execute(query)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/ext/asyncio/session.py", line 449, in execute
    result = await greenlet_spawn(
             ^^^^^^^^^^^^^^^^^^^^^
    ...<6 lines>...
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 203, in greenlet_spawn
    result = context.switch(value)
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
    return self._execute_internal(
           ~~~~~~~~~~~~~~~~~~~~~~^
        statement,
        ^^^^^^^^^^
    ...<4 lines>...
        _add_event=_add_event,
        ^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
    result: Result[Any] = compile_state_cls.orm_execute_statement(
                          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self,
        ^^^^^
    ...<4 lines>...
        conn,
        ^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
    result = conn.execute(
        statement, params or {}, execution_options=execution_options
    )
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
    return meth(
        self,
        distilled_parameters,
        execution_options or NO_OPTIONS,
    )
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
    return connection._execute_clauseelement(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self, distilled_params, execution_options
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
    ret = self._execute_context(
        dialect,
    ...<8 lines>...
        cache_hit=cache_hit,
    )
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
    return self._exec_single_context(
           ~~~~~~~~~~~~~~~~~~~~~~~~~^
        dialect, context, statement, parameters
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
    self._handle_dbapi_exception(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        e, str_statement, effective_parameters, cursor, context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 2366, in _handle_dbapi_exception
    raise exc_info[1].with_traceback(exc_info[2])
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
    ~~~~~~~~~~~~~~~~~~~~~~~^
        cursor, str_statement, effective_parameters, context
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/engine/default.py", line 952, in do_execute
    cursor.execute(statement, parameters)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 585, in execute
    self._adapt_connection.await_(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        self._prepare_and_execute(operation, parameters)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 132, in await_only
    return current.parent.switch(awaitable)  # type: ignore[no-any-return,attr-defined] # noqa: E501
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 196, in greenlet_spawn
    value = await result
            ^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 563, in _prepare_and_execute
    self._handle_exception(error)
    ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 513, in _handle_exception
    self._adapt_connection._handle_exception(error)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 799, in _handle_exception
    raise error
  File "/usr/local/lib/python3.13/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 550, in _prepare_and_execute
    self._rows = deque(await prepared_stmt.fetch(*parameters))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/asyncpg/prepared_stmt.py", line 177, in fetch
    data = await self.__bind_execute(args, 0, timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/asyncpg/prepared_stmt.py", line 268, in __bind_execute
    data, status, _ = await self.__do_execute(
                      ^^^^^^^^^^^^^^^^^^^^^^^^
        lambda protocol: protocol.bind_execute(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
            self._state, args, '', limit, True, timeout))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/site-packages/asyncpg/prepared_stmt.py", line 257, in __do_execute
    return await executor(protocol)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "asyncpg/protocol/protocol.pyx", line 205, in bind_execute
TimeoutError
I think the query used to schedule the flows is timing out, possibly due to some sort of deadlock. Other errors in the logs indicate that other queries are also timing out. How can I work around this problem? There do not appear to be any performance-related issues on my server - no other pods are using high CPU, memory, or IO.
m
Your message was too long, here's your message at the allowed limit: The following pods are running
Copy code
NAME READY STATUS RESTARTS AGE prefect-postgresql-0 1/1 Running 0 16m prefect-redis-master-0 1/1 Running 0 16m prefect-server-777f8976d5-bv7nk 1/1 Running 1 (15m ago) 16m prefect-server-background-services-cdb9d755d-9s5qx 1/1 Running 3 (15m ago) 16m
The pod 
prefect-server-background-services-cdb9d755d-9s5qx
is running 
prefect server services start
Both pods appear to be configured correctly to talk to redis 
PREFECT_SERVER_DOCKET_URL
is set correctly In the logs there is evidence that both the 
prefect-server
and 
prefect-server-background-services
pods can access redis Now I see a log error related to scheduling for the first time - it's a database timeout ``` 16 03 36 370 | ERROR | docket dependencies - ↩️ [ 27143ms] schedule_deployments(){schedule_deployments} Traceback (most recent call last) File /usr/local/lib/python3 13/site-packages/docket/worker py line 1002 in _execute result = await execution function( ^^^^^^^^^^^^^^^^^^^^^^^^^ *execution args **final_kwargs ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File /usr/local/lib/python3 13/site-packages/prefect/server/services/scheduler py line 248 in schedule_deployments result = await session execute(query) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /usr/local/lib/python3 13/site-packages/sqlalchemy/ext/asyncio/session py line 449 in execute result = await greenlet_spawn( ^^^^^^^^^^^^^^^^^^^^^ <6 lines> ) ^ File /usr/local/lib/python3 13/site-packages/sqlalchemy/util/_concurrency_py3k py line 203 in greenlet_spawn result = context switch(value) File /usr/local/lib/python3 13/site-packages/sqlalchemy/orm/session py line 2351 in execute return self _execute_internal( ~~~~~~~~~~~~~~~~~~~~~~^ statement ^^^^^^^^^^ <4 lines> _add_event=_add_event ^^^^^^^^^^^^^^^^^^^^^^ ) ^ File /usr/local/lib/python3 13/site-packages/sqlalchemy/orm/session py line 2249 in _execute_internal result Result[Any] = compile_state_cls orm_execute_statement( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ self ^^^^^ <4 lines> conn ^^^^^ ) ^ File /usr/local/lib/python3 13/site-packages/sqlalchemy/orm/context py line 306 in orm_execute_statement result = conn execute( statement params or {} execution_options=execution_options ) File /usr/local/lib/python3 13/site-packages/sqlalchemy/engine/base py line 1419 in execute return meth( self distilled_parameters execution_options or NO_OPTIONS ) File /usr/local/lib/python3 13/site-packages/sqlalchemy/sql/elements py line 527 in _execute_on_connection return connection _execute_clauseelement( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ self distilled_params execution_options ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File /usr/local/lib/python3 13/site-packages/sqlalchemy/engine/base py line 1641 in _execute_clauseelement ret = self _execute_context( dialect <8 lines> cache_hit=cache_hit ) File /usr/local/lib/python3 13/site-packages/sqlalchemy/engine/base py line 1846 in _execute_context return self _exec_single_context( ~~~~~~~~~~~~~~~~~~~~~~~~~^ dialect context statement parameters ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File /usr/local/lib/python3 13/site-packages/sqlalchemy/engine/base py line 1986 in _exec_single_context self _handle_dbapi_exception( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ e str_statement effective_parameters cursor context ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File /usr/local/lib/python3 13/site-packages/sqlalchemy/engine/base py line 2366 in _handle_dbapi_exception raise exc_info[1] with_traceback(exc_info[2]) File /usr/local/lib/python3 13/site-packages/sqlalchemy/engine/base py line 1967 in _exec_single_context self dialect do_execute( ~~~~~~~~~~~~~~~~~~~~~~~^ cursor str_statement effective_parameters context ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File /usr/local/lib/python3 13/site-packages/sqlalchemy/engine/default py line 952 in do_execute cursor execute(statement parameters) ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ File
/usr/local/lib/python3 13/site-packages/sqlalchemy/dialects/postgresql/asyncpg py line 585 in execute self _adapt_connection await_( ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ self _prepare_and_execute(operation parameters) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ) ^ File /usr/local/lib/python3 13/site-packages/sqlalchemy/util/_concurrency_py3k py line 132 in await_only return current parent switch(awaitable) # type ignore[no-any-return attr-defined] # noqa E501 ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ File /usr/local/lib/python3 13/site-packages/sqlalchemy/util/_concurrency_py3k py line 196 in greenlet_spawn value = await result ^^^^^^^^^^^^ File /usr/local/lib/python3 13/site-packages/sqlalchemy/dialects/postgresql/asyncpg py line 563 in _prepare_and_execute self _handle_exception(error) ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^ File /usr/local/lib/python3 13/site-packages/sqlalchemy/dialects/postgresql/asyncpg py line 513 in _handle_exception self _adapt_connection _handle_exception(error) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^ File /usr/local/lib/python3 13/site-packages/sqlalchemy/dialects/postgresql/asyncpg py line 799 in _handle_exception raise error File /usr/local/lib/python3 13/site-packages/sqlalchemy/dialects/postgresql/asyncpg py line 550 in _prepare_and_execute self _rows = deque(await prepared_stmt fetch(*parameters)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /usr/local/lib/python3 13/site-packages/asyncpg/prepared_stmt py line 177 in fetch data = await self __bind_execute(args 0 timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File /usr/local/lib/python3 13/site-packages/asyncpg/prepared_stmt py line 268 in __bind_execute data status _ = await self __do_execute( ^^^^^^^^^^^^^^^^^^^^^^^^ lambda protocol protocol
n
hi this looks like a statement timeout from postgres
how big is your db? the events table, for example
j
Hi @Nate, thanks for your help - I can see the statement in the prefect logs. I believe it is this one:
Copy code
2026-05-27 16:03:36.340 GMT [159] STATEMENT:  SELECT deployment.id
	FROM deployment
	WHERE deployment.paused IS NOT true AND (EXISTS (SELECT $1::INTEGER AS anon_1
	FROM deployment_schedule
	WHERE deployment_schedule.deployment_id = deployment.id AND deployment_schedule.active IS true AND ((SELECT count(*) AS count_1
	FROM flow_run
	WHERE flow_run.deployment_id = deployment_schedule.deployment_id AND flow_run.state_type = $2::state_type AND flow_run.next_scheduled_start_time >= $3::TIMESTAMP WITH TIME ZONE AND flow_run.auto_scheduled IS true AND CAST((flow_run.created_by ->> $4::TEXT) AS VARCHAR) = CAST(deployment_schedule.id AS VARCHAR)) < $5::INTEGER OR (SELECT max(flow_run.next_scheduled_start_time) AS max_1
	FROM flow_run
	WHERE flow_run.deployment_id = deployment_schedule.deployment_id AND flow_run.state_type = $6::state_type AND flow_run.next_scheduled_start_time >= $7::TIMESTAMP WITH TIME ZONE AND flow_run.auto_scheduled IS true AND CAST((flow_run.created_by ->> $4::TEXT) AS VARCHAR) = CAST(deployment_schedule.id AS VARCHAR)) < $8::TIMESTAMP WITH TIME ZONE))) ORDER BY deployment.id
	 LIMIT $9::INTEGER
2026-05-27 16:03:50.484 GMT [2716] ERROR:  canceling statement due to user request
The logs indicate a 15s timeout, which I've now increased to 60s.
Copy code
Edit: this is not what the logs indicate - the statement follows "ERROR:  canceling statement due to user request"
Having deleted my prefect installation and redeployed it, my flows are now being scheduled. Some flows started being scheduled yesterday after I did this, then stopped. I'll post here if the flows stop being scheduled. The size of the events table is 108634 rows, 208MB. The size of the deployment table is 49 rows, 192kB The size of the flow table is 574 rows, 688kB
n
sounds good!
j
Ok - my prefect has stopped scheduling new flow runs, and this is the latest 
created
time on any flow run: 2026-05-27 200221.219929+00. Info from logs: At 20:03, there's about a minute of network errors in the background services pod logs caused by failing DNS lookups when connecting to redis. Around 20:17, there are lots of network errors where it fails to connect to postgres. At 200428, the prefect-worker pod got restarted, then again at 201915. I don't have the logs from before 200428. I'm also seeing lots of network errors in the prefect-server pod. The postgres pod was shut down at 201351 then restarted, but there's nothing interesting in the logs around 2002 2004. The proximal cause would appear to be flaky networking in my k8s cluster, but I'd expect the flow run scheduling to survive that.
I think I've been able to replicate the problem when running 
prefect server services start
locally with some breakpoints. If redis gets restarted, then 
schedule_deployments
stops being called.
I can see some related issues which were addressed recently: • https://github.com/PrefectHQ/prefect/issues/20690#event-22839475292 • https://github.com/PrefectHQ/prefect/issues/19080 but my prefect version is 3.7.2, after those were fixed. The docket version is 0.20.3, and the prefect-redis version is 0.2.11.
Open in Slack
PreviousNext
Powered by