https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I have issues with my tests. I had th...
# ask-marvin
k
@Marvin I have issues with my tests. I had them running, but after refactoring, it seems the test prefect server won't run:
Copy code
@pytest.fixture(autouse=True, scope="session")
def prefect_test_fixture():
    with prefect_test_harness():
        yield


@pytest.mark.asyncio
async def test_trigger_flow_fns_with_full_payload_works_determinstically(mocker):
   ...

    result = await my_flow(
        {
            ...
        }
    )

    assert ...
Now, when starting, there's a server timeout:
Copy code
============================== 1 error in 32.50s ===============================
--- Logging error ---
Traceback (most recent call last):
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/logging/handlers.py", line 355, in emit
    self.console.print(message, soft_wrap=True)
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 1697, in print
    with self:
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 870, in __exit__
    self._exit_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 826, in _exit_buffer
    self._check_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 2038, in _check_buffer
    self._write_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 2107, in _write_buffer
    self.file.write(text)
ValueError: I/O operation on closed file.
Call stack:
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/server/api/server.py", line 960, in stop
    <http://subprocess_server_logger.info|subprocess_server_logger.info>(
  File "/usr/lib/python3.10/logging/__init__.py", line 1477, in info
    self._log(INFO, msg, args, **kwargs)
  File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log
    self.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle
    self.emit(record)
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/logging/handlers.py", line 361, in emit
    self.handleError(record)
Message: 'Stopping temporary server on <http://127.0.0.1:8230>'
Arguments: ()
Finished running tests!
m
✅ thought for 93.3 seconds
Thanks for the details — this looks like two things happening at once: 1) The temporary Prefect API server started by 
prefect_test_harness()
is timing out at startup (default is 30s). 2) During teardown, Prefect’s Rich console logger tries to print “Stopping temporary server...” but pytest has already closed the captured output stream, leading to the “I/O operation on closed file” logging error. That logging error is noisy but secondary; the root issue is the server startup timeout. Quick fixes to try - Give the server more time to start and disable rich console during tests to avoid the teardown logging error:
Copy code
# conftest.py
import os
import pytest
from prefect.testing.utilities import prefect_test_harness

@pytest.fixture(scope="session", autouse=True)
def prefect_test_server():
    # Prevent Rich from writing to a closed stream during pytest teardown
    os.environ.setdefault("PREFECT_LOGGING_RICH_CONSOLE", "false")
    # Optional: get more diagnostics while debugging
    # os.environ.setdefault("PREFECT_LOGGING_LEVEL", "DEBUG")

    with prefect_test_harness(server_startup_timeout=60):
        yield
- Verify no other fixture or environment variable is also starting an ephemeral/hosted server (e.g. 
enable_ephemeral_server
, 
hosted_api_server
, or a 
PREFECT_API_URL
you set elsewhere). Running two servers at once can cause port conflicts or unexpected state. - Ensure port 8230 (the default in your log) isn’t already in use by another process on your machine/CI. If it is, kill that process or ensure your test harness is the one that controls the server lifecycle. Sanity check the server Add a tiny smoke test to confirm the API is reachable once the fixture is in place:
Copy code
import anyio
from prefect.client.orchestration import get_client

def test_prefect_api_smoke():
    async def ping():
        async with get_client() as c:
            await c.read_healthcheck()
    anyio.run(ping)
If this passes, the server is up — any remaining failures are likely test-specific. A couple of optional refinements - If your tests do not need the API (e.g., you’re just calling flows directly), you can skip starting the server entirely and run offline by ensuring 
PREFECT_API_URL
is unset or empty for tests. - If you need per-test isolation of API state, you can make the fixture function-scoped instead of session-scoped (slower but avoids some teardown ordering issues in certain environments). Questions to help pinpoint the cause - Which Prefect version are you on? (Assuming 3.x here.) - Did your refactor add any client interactions (e.g., deployments, reading/writing blocks) that require the API for imports or test collection? - Are you using any other logging/capture customizations or pytest plugins that might close stdout/stderr early? References - Prefect 3 testing guide: Test workflows - Prefect testing utilities API (incl. 
prefect_test_harness
): prefect.testing utilities and fixtures - Example conftest usage: Sample test setup with Prefect harness If you share your 
conftest.py
and any logging/pytest config that changed in your refactor, I can tailor this further.
k
@Marvin well, I'm just running the smoke test now, and it's still the same issue. I'm running prefect 3.x
Copy code
@pytest.fixture(scope="session", autouse=True)
def prefect_test_server():
    os.environ.setdefault("PREFECT_LOGGING_RICH_CONSOLE", "false")
    os.environ.setdefault("PREFECT_LOGGING_LEVEL", "DEBUG")
    with prefect_test_harness(server_startup_timeout=60):
        yield


def test_prefect_api_smoke():
    async def ping():
        async with get_client() as c:
            await c.read_healthcheck()  # pyright: ignore[reportAttributeAccessIssue]

    anyio.run(ping)
gives:
Copy code
========================= 1 error in 62.94s (0:01:02) ==========================
--- Logging error ---
Traceback (most recent call last):
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/logging/handlers.py", line 355, in emit
    self.console.print(message, soft_wrap=True)
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 1697, in print
    with self:
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 870, in __exit__
    self._exit_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 826, in _exit_buffer
    self._check_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 2038, in _check_buffer
    self._write_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 2107, in _write_buffer
    self.file.write(text)
ValueError: I/O operation on closed file.
Call stack:
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/server/api/server.py", line 960, in stop
    subprocess_server_logger.info(
  File "/usr/lib/python3.10/logging/__init__.py", line 1477, in info
    self._log(INFO, msg, args, **kwargs)
  File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log
    self.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle
    self.emit(record)
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/logging/handlers.py", line 361, in emit
    self.handleError(record)
Message: 'Stopping temporary server on <http://127.0.0.1:8224>'
Arguments: ()
Finished running tests!
So it seems the server just can't start?
m
✅ thought for 188.6 seconds
Yes — based on the timeout and the “Stopping temporary server…” message, the ephemeral Prefect API server that 
prefect_test_harness()
spawns is not coming up within the timeout window. The Rich logging error at teardown is just noise because pytest has already closed the captured stream. Let’s narrow this down and get you unblocked. Do these 5 things in order 1) Make logging/env settings take effect before Prefect is imported Environment flags set inside the fixture are too late for Prefect’s logging setup. Move these to the very top of conftest.py, before any Prefect imports:
Copy code
# conftest.py (top of file, before any imports)
import os
os.environ["PREFECT_LOGGING_RICH_CONSOLE"] = "false"
os.environ["PREFECT_LOGGING_TO_CONSOLE"] = "false"]  # avoid console entirely during tests
os.environ["PREFECT_LOGGING_TO_FILE"] = "true"]
os.environ["PREFECT_SERVER_LOGGING_LEVEL"] = "DEBUG"  # server subprocess logs
os.environ["PREFECT_UI_ENABLED"] = "false"            # no UI during tests
# Optionally: isolate state to a temp dir to avoid permissions/corruption
# import tempfile; os.environ["PREFECT_HOME"] = tempfile.mkdtemp(prefix="prefect-test-")
2) Strengthen your harness fixture - Ensure no external API URL is inherited. - Give a longer startup timeout. - Optionally isolate Prefect home to a temp directory (sometimes stale state/permissions cause startup failures).
Copy code
import os
import pytest
from prefect.testing.utilities import prefect_test_harness

@pytest.fixture(scope="session", autouse=True)
def prefect_test_server():
    # Ensure we don’t point at Cloud or any external API during tests
    os.environ.pop("PREFECT_API_URL", None)

    # If you didn’t set PREFECT_HOME at module import time, do it here:
    # import tempfile; os.environ.setdefault("PREFECT_HOME", tempfile.mkdtemp(prefix="prefect-test-"))

    with prefect_test_harness(server_startup_timeout=120):
        yield
3) Quick diagnostics inside pytest Add this tiny test to confirm what settings are actually active during the run:
Copy code
from prefect.settings import get_current_settings

def test_prefect_settings_dump():
    s = get_current_settings()
    print("API URL:", s.api_url)
    print("HOME:", s.home)
    print("LOG TO CONSOLE:", s.logging_to_console)
- Expected: 
API URL
should be a local http://127.0.0.1:&lt;port> set by the harness. - If it’s empty or pointing at Prefect Cloud, something is overriding the harness. 4) Try the server outside pytest From the same virtualenv:
Copy code
prefect server start --no-ui --log-level DEBUG
- If this fails to start or hangs, we’ve isolated an environment-level issue (permissions, ports, conflicting env vars, or a bug). Capture and share the output. - If this starts cleanly and shows an API URL, the problem is specific to pytest startup/teardown or parallelization. 5) Check for parallelization or port conflicts - Are you using pytest-xdist (
-n auto
)? Each worker may try to start its own server and compete for ports. Disable xdist during this debug or ensure each worker has its own 
PREFECT_HOME
and free port. - Ensure nothing else is binding to 127.0.0.1 on ports in the 8200–8300 range. Alternative fixture if the harness keeps failing You can try the built-in hosted server fixtures:
Copy code
# conftest.py
import pytest
from prefect.testing.fixtures import use_hosted_api_server

@pytest.fixture(scope="session", autouse=True)
def _use_hosted_server(use_hosted_api_server):
    # This fixture starts a hosted API server and sets PREFECT_API_URL for tests
    yield
This follows the same pattern the harness relies on but sometimes behaves better if there’s odd state in the test environment. If you need to run fully offline (no API), skip the server If your tests don’t hit the API, avoid starting it: - Ensure 
PREFECT_API_URL
is empty/unset for tests. - Call flows/tasks directly and avoid client interactions. If the problem persists, please share: - Output of 
prefect version
- Your OS/CI environment
- Whether you’re using pytest-xdist - The full output of 
prefect server start --no-ui --log-level DEBUG
run manually References - Test workflows: Prefect 3 test workflows - Testing utilities and fixtures (incl. 
prefect_test_harness
): prefect.testing utilities and fixtures
k
@Marvin It's really about starting from the code. Running the test_prefect_settings_dump:
Copy code
:test_prefect_settings_dump
========================= 1 error in 62.79s (0:01:02) ==========================
--- Logging error ---
Traceback (most recent call last):
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/logging/handlers.py", line 355, in emit
    self.console.print(message, soft_wrap=True)
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 1697, in print
    with self:
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 870, in __exit__
    self._exit_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 826, in _exit_buffer
    self._check_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 2038, in _check_buffer
    self._write_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 2107, in _write_buffer
    self.file.write(text)
ValueError: I/O operation on closed file.
Call stack:
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/server/api/server.py", line 960, in stop
    <http://subprocess_server_logger.info|subprocess_server_logger.info>(
  File "/usr/lib/python3.10/logging/__init__.py", line 1477, in info
    self._log(INFO, msg, args, **kwargs)
  File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log
    self.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle
    self.emit(record)
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/logging/handlers.py", line 361, in emit
    self.handleError(record)
Message: 'Stopping temporary server on <http://127.0.0.1:8459>'
also, starting from command line looks ok
Copy code
prefect server start --no-ui --log-level DEBUG

 ___ ___ ___ ___ ___ ___ _____
| _ \ _ \ __| __| __/ __|_   _|
|  _/   / _|| _|| _| (__  | |
|_| |_|_\___|_| |___\___| |_|

Configure Prefect to communicate with the server with:

    prefect config set PREFECT_API_URL=<http://127.0.0.1:4200/api>

View the API reference documentation at <http://127.0.0.1:4200/docs>

The dashboard is disabled. Set `PREFECT_UI_ENABLED=1` to re-enable it.
m
✅ thought for 80.3 seconds
Thanks — the fact that 
prefect server start
works but the harness still times out points to an interaction with pytest startup/teardown or environment setup, not a general server issue. A few targeted steps to isolate and fix this: 1) Set env before any Prefect import These must be at the very top of conftest.py (before importing Prefect or your code) so the server subprocess and logging pick them up:
Copy code
# conftest.py (top of file)
import os
os.environ["PREFECT_LOGGING_RICH_CONSOLE"] = "0"
os.environ["PREFECT_LOGGING_TO_CONSOLE"] = "0"
os.environ["PREFECT_LOGGING_TO_FILE"] = "1"
os.environ["PREFECT_SERVER_LOGGING_LEVEL"] = "DEBUG"
os.environ["PREFECT_UI_ENABLED"] = "0"
# Optional: force a known-free port and host, mirroring your working CLI setup
os.environ["PREFECT_SERVER_API_HOST"] = "127.0.0.1"
os.environ["PREFECT_SERVER_API_PORT"] = "4201"
# Strongly recommended: isolate Prefect state/logs to a temp directory for tests
# import tempfile; os.environ["PREFECT_HOME"] = tempfile.mkdtemp(prefix="prefect-test-")
# Ensure we never inherit a Cloud URL
os.environ.pop("PREFECT_API_URL", None)
2) Try the hosted server fixture as a control If this works, the issue is specific to 
prefect_test_harness()
in your environment.
Copy code
# conftest.py (after the env setup above)
import pytest
from prefect.testing.fixtures import use_hosted_api_server

@pytest.fixture(scope="session", autouse=True)
def _api(use_hosted_api_server):
    # Starts an ephemeral server and sets PREFECT_API_URL for the session
    yield
Then the smoke test:
Copy code
def test_prefect_api_smoke():
    # Confirm we’re pointed at the ephemeral server
    from prefect.settings import get_current_settings
    s = get_current_settings()
    assert s.api_url, f"Missing API URL: {s}"

    import anyio
    from prefect.client.orchestration import get_client
    async def ping():
        async with get_client() as c:
            await c.read_healthcheck()
    anyio.run(ping)
3) If you must use prefect_test_harness, harden it If the control above works, bring back the harness with stronger isolation. The key bits: env at module import time + longer timeout + isolated HOME.
Copy code
# conftest.py (after the env setup above)
import pytest
from prefect.testing.utilities import prefect_test_harness

@pytest.fixture(scope="session", autouse=True)
def prefect_test_server():
    with prefect_test_harness(server_startup_timeout=120):
        yield
4) Capture server logs to see why readiness never happens With 
PREFECT_LOGGING_TO_FILE=1
and an isolated 
PREFECT_HOME
, Prefect writes to PREFECT_HOME/logs/prefect.log. After the timeout, check that file for server subprocess logs (at DEBUG) — it often shows the exact reason (e.g., DB lock, port bind error, import error in the subprocess). 5) Quick sanity checks - Disable pytest parallelization while debugging (if using xdist): run without 
-n
. - Try running pytest with 
-s
once (disables capture) to see if the Rich/console teardown noise goes away. It won’t fix readiness, but it’ll make logs cleaner. - Since the CLI used port 4200 successfully, forcing a nearby port (4201) avoids random high-port firewall or corporate policy oddities. If the hosted fixture also times out, or if the server log shows something odd, please share: - The exact output of 
prefect version
- Your OS/CI environment - Whether you’re using pytest-xdist - The last 50–100 lines of PREFECT_HOME/logs/prefect.log after the failure Links - Test workflows: Prefect 3 test workflows - Testing utilities and fixtures (incl. 
prefect_test_harness
, 
use_hosted_api_server
): prefect.testing utilities and fixtures Once we see the server log, we should be able to pinpoint why readiness never completes.
k
@Marvin ok, let's take one step back: I'm just running 1 test. I expect the 
prefect_test_harness
command to start up a server. That fails. That's where it already goes wrong. No need to start talking about other stuff (like parallelization etc). The server just deosn't start
Copy code
$ prefect version
Version:              3.4.22
API version:          0.8.4
Python version:       3.10.12
Git commit:           6a6d114f
Built:                Fri, Oct 03, 2025 06:15 PM
OS/Arch:              linux/x86_64
Profile:              local
Server type:          ephemeral
Pydantic version:     2.11.9
Server:
  Database:           sqlite
  SQLite version:     3.37.2
Integrations:
  prefect-kubernetes: 0.6.5
  prefect-docker:     0.6.6
m
✅ thought for 77.2 seconds
Got it — let’s focus purely on “prefect_test_harness should start a server, but it times out.” Please try this minimal, pytest-free script. It pins host/port, isolates state, and writes logs to a temp directory so we can see exactly why readiness never completes:
Copy code
# run_me.py
import os, tempfile, anyio

# Isolate Prefect state/logs to a fresh temp dir
os.environ["PREFECT_HOME"] = tempfile.mkdtemp(prefix="prefect-test-")

# Make the server environment explicit and quiet
os.environ["PREFECT_SERVER_API_HOST"] = "127.0.0.1"
os.environ["PREFECT_SERVER_API_PORT"] = "4201"
os.environ["PREFECT_UI_ENABLED"] = "0"
os.environ["PREFECT_LOGGING_TO_FILE"] = "1"
os.environ["PREFECT_LOGGING_TO_CONSOLE"] = "0"
os.environ.pop("PREFECT_API_URL", None)

from prefect.testing.utilities import prefect_test_harness
from prefect.client.orchestration import get_client

with prefect_test_harness(server_startup_timeout=120):
    async def main():
        async with get_client() as c:
            await c.read_healthcheck()
    anyio.run(main)

print("OK")
- If this succeeds: the harness works in your environment; the pytest run is interfering (we can then fix pytest-specific setup). - If this still times out: please open the log file in the temp PREFECT_HOME you see printed by the script, e.g. 
<temp>/logs/prefect.log
, and share the last ~100 lines. That log contains the server subprocess messages at startup and will tell us exactly why readiness never happens (DB lock, bind error, migration issue, etc.). If you want to try a code-only alternative right away (still “start a server from code”), use the built-in fixture that enables the ephemeral server directly:
Copy code
# conftest.py (minimal)
import os
os.environ["PREFECT_UI_ENABLED"] = "0"
os.environ["PREFECT_LOGGING_TO_CONSOLE"] = "0"
os.environ.pop("PREFECT_API_URL", None)

import pytest

# This starts Prefect's ephemeral server for the session
@pytest.fixture(scope="session", autouse=True)
def _server(enable_ephemeral_server):
    yield
Then your smoke test should pass:
Copy code
def test_prefect_api_smoke():
    import anyio
    from prefect.client.orchestration import get_client
    async def ping():
        async with get_client() as c:
            await c.read_healthcheck()
    anyio.run(ping)
But the key next step to debug the harness timeout is the temp PREFECT_HOME + file logs. Once we see those logs, we’ll pinpoint the cause quickly.
k
@Marvin what about this:
Copy code
12:09:29.791 | DEBUG   | prefect.server.services.proactivetriggers - About to run ProactiveTriggers...
12:09:29.791 | DEBUG   | prefect.server.services.proactivetriggers - Finished running ProactiveTriggers. Next run at 2025-10-04 12:09:29.791358+02:00
12:09:29.791 | DEBUG   | prefect.server.services.proactivetriggers - About to run ProactiveTriggers...
12:09:29.791 | DEBUG   | prefect.server.services.proactivetriggers - Finished running ProactiveTriggers. Next run at 2025-10-04 12:09:29.791710+02:00
12:09:29.791 | DEBUG   | prefect.server.services.proactivetriggers - About to run ProactiveTriggers...
12:09:29.792 | DEBUG   | prefect.server.services.proactivetriggers - Finished running ProactiveTriggers. Next run at 2025-10-04 12:09:29.792056+02:00
12:09:29.792 | DEBUG   | prefect.server.services.proactivetriggers - About to run ProactiveTriggers...
12:09:29.792 | DEBUG   | prefect.server.services.proactivetriggers - Finished running ProactiveTriggers. Next run at 2025-10-04 12:09:29.792406+02:00
12:09:29.792 | DEBUG   | prefect.server.services.proactivetriggers - About to run ProactiveTriggers...
12:09:29.792 | DEBUG   | prefect.server.services.proactivetriggers - Finished running ProactiveTriggers. Next run at 2025-10-04 12:09:29.792750+02:00
12:09:29.792 | DEBUG   | prefect.server.services.proactivetriggers - About to run ProactiveTriggers...
12:09:29.793 | DEBUG   | prefect.server.services.proactivetriggers - Finished running ProactiveTriggers. Next run at 2025-10-04 12:09:29.793098+02:00
12:09:29.793 | DEBUG   | prefect.server.services.proactivetriggers - About to run ProactiveTriggers...
12:09:29.793 | DEBUG   | prefect.server.services.proactivetriggers - Finished running ProactiveTriggers. Next run at 2025-10-04 12:09:29.793449+02:00
12:09:29.793 | DEBUG   | prefect.server.services.proactivetriggers - About to run ProactiveTriggers...
--- Logging error ---
Traceback (most recent call last):
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/logging/handlers.py", line 355, in emit
    self.console.print(message, soft_wrap=True)
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 1697, in print
    with self:
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 870, in __exit__
    self._exit_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 826, in _exit_buffer
    self._check_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 2038, in _check_buffer
    self._write_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 2107, in _write_buffer
    self.file.write(text)
ValueError: I/O operation on closed file.
Call stack:

------------------------------ Captured log setup ------------------------------
INFO     prefect:server.py:881 Starting temporary server on <http://127.0.0.1:8719>
See <https://docs.prefect.io/v3/concepts/server#how-to-guides> for more information on running a dedicated Prefect server.
=========================== short test summary info ============================
ERROR python/fraio/fraio_orchestrator/tests/fraio_orchestrator/test_fraio_trigger_flow.py::test_prefect_settings_dump
========================= 1 error in 62.96s (0:01:02) ==========================
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/server/api/server.py", line 960, in stop
    <http://subprocess_server_logger.info|subprocess_server_logger.info>(
  File "/usr/lib/python3.10/logging/__init__.py", line 1477, in info
    self._log(INFO, msg, args, **kwargs)
  File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log
    self.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle
    self.emit(record)
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/logging/handlers.py", line 361, in emit
    self.handleError(record)
Message: 'Stopping temporary server on <http://127.0.0.1:8719>'
Arguments: ()
Finished running tests!
m
Your message was too long, here's your message at the allowed limit: what about this ``` 12 09 29 791 | DEBUG | prefect server services proactivetriggers - About to run ProactiveTriggers 12 09 29 791 | DEBUG | prefect server services proactivetriggers - Finished running ProactiveTriggers Next run at 2025-10-04 12 09 29 791358+02 00 12 09 29 791 | DEBUG | prefect server services proactivetriggers - About to run ProactiveTriggers 12 09 29 791 | DEBUG | prefect server services proactivetriggers - Finished running ProactiveTriggers Next run at 2025-10-04 12 09 29 791710+02 00 12 09 29 791 | DEBUG | prefect server services proactivetriggers - About to run ProactiveTriggers 12 09 29 792 | DEBUG | prefect server services proactivetriggers - Finished running ProactiveTriggers Next run at 2025-10-04 12 09 29 792056+02 00 12 09 29 792 | DEBUG | prefect server services proactivetriggers - About to run ProactiveTriggers 12 09 29 792 | DEBUG | prefect server services proactivetriggers - Finished running ProactiveTriggers Next run at 2025-10-04 12 09 29 792406+02 00 12 09 29 792 | DEBUG | prefect server services proactivetriggers - About to run ProactiveTriggers 12 09 29 792 | DEBUG | prefect server services proactivetriggers - Finished running ProactiveTriggers Next run at 2025-10-04 12 09 29 792750+02 00 12 09 29 792 | DEBUG | prefect server services proactivetriggers - About to run ProactiveTriggers 12 09 29 793 | DEBUG | prefect server services proactivetriggers - Finished running ProactiveTriggers Next run at 2025-10-04 12 09 29 793098+02 00 12 09 29 793 | DEBUG | prefect server services proactivetriggers - About to run ProactiveTriggers 12 09 29 793 | DEBUG | prefect server services proactivetriggers - Finished running ProactiveTriggers Next run at 2025-10-04 12 09 29 793449+02 00 12 09 29 793 | DEBUG | prefect server services proactivetriggers - About to run ProactiveTriggers --- Logging error --- Traceback (most recent call last) File /home/vintecc/py310/lib/python3 10/site-packages/prefect/logging/handlers py line 355 in emit self console print(message soft_wrap=True) File /home/vintecc/py310/lib/python3 10/site-packages/rich/console py line 1697 in print with self File /home/vintecc/py310/lib/python3 10/site-packages/rich/console py line 870 in exit self _exit_buffer() File /home/vintecc/py310/lib/python3 10/site-packages/rich/console py line 826 in _exit_buffer self _check_buffer() File /home/vintecc/py310/lib/python3 10/site-packages/rich/console py line 2038 in _check_buffer self _write_buffer() File /home/vintecc/py310/lib/python3 10/site-packages/rich/console py line 2107 in _write_buffer self file write(text) ValueError I/O operation on closed file Call stack ------------------------------ Captured log setup ------------------------------ INFO prefect server py 881 Starting temporary server on <http //127 0 0 1 8719> See <https //docs prefect io/v3/concepts/server#how-to-guides> for more information on running a dedicated Prefect server =========================== short test summary info ============================ ERROR python/fraio/fraio_orchestrator/tests/fraio_orchestrator/test_fraio_trigger_flow py test_prefect_settings_dump ========================= 1 error in 62 96s (0 01 02) ========================== File /home/vintecc/py310/lib/python3 10/site-packages/prefect/server/api/server py line 960 in stop <http //subprocess_server_logger info|subprocess_server_logger info>( File /usr/lib/python3 10/logging/__init__ py line 1477 in info self _log(INFO msg args **kwargs) File /usr/lib/python3 10/logging/__init__ py line 1624 in _log self handle(record) File /usr/lib/python3 10/logging/__init__ py line 1634 in handle self callHandlers(record) File /usr/lib/python3 10/logging/__init__ py line 1696 in callHandlers hdlr handle(record) File /usr/lib/python3 10/logging/__init__ py line 968 in handle self emit(record) File
k
@Marvin ok:
Copy code
12:09:29.793 | DEBUG   | prefect.server.services.proactivetriggers - Finished running ProactiveTriggers. Next run at 2025-10-04 12:09:29.793098+02:00
12:09:29.793 | DEBUG   | prefect.server.services.proactivetriggers - About to run ProactiveTriggers...
12:09:29.793 | DEBUG   | prefect.server.services.proactivetriggers - Finished running ProactiveTriggers. Next run at 2025-10-04 12:09:29.793449+02:00
12:09:29.793 | DEBUG   | prefect.server.services.proactivetriggers - About to run ProactiveTriggers...
--- Logging error ---
Traceback (most recent call last):
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/logging/handlers.py", line 355, in emit
    self.console.print(message, soft_wrap=True)
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 1697, in print
    with self:
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 870, in __exit__
    self._exit_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 826, in _exit_buffer
    self._check_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 2038, in _check_buffer
    self._write_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 2107, in _write_buffer
    self.file.write(text)
ValueError: I/O operation on closed file.
Call stack:

------------------------------ Captured log setup ------------------------------
INFO     prefect:server.py:881 Starting temporary server on <http://127.0.0.1:8719>
See <https://docs.prefect.io/v3/concepts/server#how-to-guides> for more information on running a dedicated Prefect server.
=========================== short test summary info ============================
ERROR python/fraio/fraio_orchestrator/tests/fraio_orchestrator/test_fraio_trigger_flow.py::test_prefect_settings_dump
========================= 1 error in 62.96s (0:01:02) ==========================
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/server/api/server.py", line 960, in stop
    subprocess_server_logger.info(
  File "/usr/lib/python3.10/logging/__init__.py", line 1477, in info
    self._log(INFO, msg, args, **kwargs)
  File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log
    self.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle
    self.emit(record)
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/logging/handlers.py", line 361, in emit
    self.handleError(record)
Message: 'Stopping temporary server on <http://127.0.0.1:8719>'
Arguments: ()
Finished running tests!
m
✅ thought for 41.9 seconds
Thanks for the log — that’s the clue we needed. - Your log shows Prefect services (e.g., ProactiveTriggers) running, and “Starting temporary server on http://127.0.0.1:8719”, which means the server subprocess DID start. - The harness times out because its readiness check can’t reach that URL from the parent process. - The Rich “I/O operation on closed file” is just teardown noise after the timeout. The most common reason: a system HTTP(S) proxy is intercepting localhost requests. If `HTTP_PROXY`/`HTTPS_PROXY`/`ALL_PROXY` is set and 
NO_PROXY
does not include 127.0.0.1/localhost, the harness’s health-check (and your 
get_client()
ping) will route to the proxy and hang for ~60s. Fix: disable proxies for localhost during tests Put this at the very top of conftest.py (before any Prefect imports), or in your minimal script before importing Prefect:
Copy code
import os

# Disable proxies during tests
for k in ("HTTP_PROXY","HTTPS_PROXY","ALL_PROXY","http_proxy","https_proxy","all_proxy"):
    os.environ.pop(k, None)

# Ensure localhost bypasses any proxy that might still be injected by the environment
os.environ["NO_PROXY"] = "127.0.0.1,localhost"
os.environ["no_proxy"] = "127.0.0.1,localhost"

# Optional: make server binding explicit and quiet logging to console
os.environ["PREFECT_SERVER_API_HOST"] = "127.0.0.1"
os.environ["PREFECT_SERVER_API_PORT"] = "4201"
os.environ["PREFECT_UI_ENABLED"] = "0"
os.environ["PREFECT_LOGGING_TO_CONSOLE"] = "0"
os.environ["PREFECT_LOGGING_TO_FILE"] = "1"
os.environ.pop("PREFECT_API_URL", None)
Then use your original harness fixture:
Copy code
import pytest
from prefect.testing.utilities import prefect_test_harness

@pytest.fixture(scope="session", autouse=True)
def prefect_test_server():
    with prefect_test_harness(server_startup_timeout=120):
        yield
And the smoke test:
Copy code
def test_prefect_api_smoke():
    import anyio
    from prefect.client.orchestration import get_client
    async def ping():
        async with get_client() as c:
            await c.read_healthcheck()
    anyio.run(ping)
Why this works - The server is already starting; the failure is the client (harness health-check and your test) trying to reach 127.0.0.1 via a proxy. Setting NO_PROXY and unsetting proxy vars lets the health-check hit the local server directly. If this still times out after unsetting proxies, let me know and please share the last 50–100 lines of the Prefect log again. But based on your logs, the proxy issue fits perfectly.
k
@Marvin ok, so we're having progress. What's the
Copy code
12:13:06.221 | DEBUG   | prefect.server.services.proactivetriggers - About to run ProactiveTriggers...
12:13:06.222 | DEBUG   | prefect.server.services.proactivetriggers - Finished running ProactiveTriggers. Next run at 2025-10-04 12:13:06.222052+02:00
about?
m
✅ thought for 28.4 seconds
Those are just server-side background service logs. In Prefect 3.x, the server runs a few loop services; one of them is ProactiveTriggers. - What it is: a service that periodically evaluates “proactive” event triggers—rules that should fire when some expected event does not occur within a time window (the “within” period). - Why you see the messages: each evaluation cycle logs “About to run ProactiveTriggers…”, does a pass over any configured triggers, then logs “Finished… Next run at …” with the next scheduled tick. It’s normal noise at DEBUG level and not related to your startup problem. If you don’t use event triggers, you can ignore those messages. If you want fewer logs during tests, keep logging to console disabled or set log levels higher. For reference: - Concepts: Event triggers - Services API (includes triggers service): Server events services
k
@Marvin ok, so let's try again where we're at:
Copy code
os.environ["PREFECT_SERVER_LOGGING_LEVEL"] = "INFO"
os.environ["PREFECT_UI_ENABLED"] = "0"
# Optional: force a known-free port and host, mirroring your working CLI setup
os.environ["PREFECT_SERVER_API_HOST"] = "127.0.0.1"
os.environ["PREFECT_SERVER_API_PORT"] = "4201"
# Strongly recommended: isolate Prefect state/logs to a temp directory for tests
import tempfile

os.environ["PREFECT_HOME"] = tempfile.mkdtemp(prefix="prefect-test-")
# Ensure we never inherit a Cloud URL
os.environ.pop("PREFECT_API_URL", None)


@pytest.fixture(scope="session", autouse=True)
def prefect_test_server():
    print(f"prefect home: {os.environ['PREFECT_HOME']}")

    with prefect_test_harness(server_startup_timeout=60):
        yield


def test_prefect_settings_dump():
    s = get_current_settings()
    print("API URL:", s.api_url)
    print("HOME:", s.home)
    print("LOG TO CONSOLE:", s.logging_to_console)
results in:
Copy code
============================= test session starts ==============================
platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.6.0
rootdir: /workspace
configfile: pytest.ini
plugins: devtools-0.12.2, docker-3.2.3, anyio-4.11.0, asyncio-0.23.8, xdist-3.8.0, postgresql-7.0.2, dash-3.2.0, hydra-core-1.3.2, Faker-37.8.0, cov-6.2.1
asyncio: mode=strict
collected 1 item

python/fraio/fraio_orchestrator/tests/fraio_orchestrator/test_fraio_trigger_flow.py E [100%]

==================================== ERRORS ====================================
_________________ ERROR at setup of test_prefect_settings_dump _________________

    @pytest.fixture(scope="session", autouse=True)
    def prefect_test_server():
        print(f"prefect home: {os.environ['PREFECT_HOME']}")
    
>       with prefect_test_harness(server_startup_timeout=60):

python/fraio/fraio_orchestrator/tests/fraio_orchestrator/test_fraio_trigger_flow.py:43: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3.10/contextlib.py:135: in __enter__
    return next(self.gen)
/home/vintecc/py310/lib/python3.10/site-packages/prefect/testing/utilities.py:168: in prefect_test_harness
    test_server.start(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <prefect.server.api.server.SubprocessASGIServer object at 0x7787051b7100>
timeout = 60

    def start(self, timeout: Optional[int] = None) -> None:
        """
        Start the server in a separate process. Safe to call multiple times; only starts
        the server once.
    
        Args:
            timeout: The maximum time to wait for the server to start
        """
        if not self.running:
            if self.port is None:
                self.port = self.find_available_port()
            assert self.port is not None, "Port must be provided or available"
            help_message = (
                f"Starting temporary server on {self.address}\nSee "
                "<https://docs.prefect.io/v3/concepts/server#how-to-guides> "
                "for more information on running a dedicated Prefect server."
            )
            <http://subprocess_server_logger.info|subprocess_server_logger.info>(help_message)
            try:
                self.running = True
                self.server_process = self._run_uvicorn_command()
                atexit.register(self.stop)
                with httpx.Client() as client:
                    response = None
                    elapsed_time = 0
                    max_wait_time = (
                        timeout
                        or PREFECT_SERVER_EPHEMERAL_STARTUP_TIMEOUT_SECONDS.value()
                    )
                    while elapsed_time < max_wait_time:
                        if self.server_process.poll() == 3:
                            self.port = self.find_available_port()
                            self.server_process = self._run_uvicorn_command()
                            continue
                        try:
                            response = client.get(f"{self.api_url}/health")
                        except httpx.ConnectError:
                            pass
                        else:
                            if response.status_code == 200:
                                break
                        time.sleep(0.1)
                        elapsed_time += 0.1
                    if response:
                        response.raise_for_status()
                    if not response:
                        error_message = "Timed out while attempting to connect to ephemeral Prefect API server."
                        if self.server_process.poll() is not None:
                            error_message += f" Ephemeral server process exited with code {self.server_process.returncode}."
                        if self.server_process.stdout:
                            error_message += (
                                f" stdout: {self.server_process.stdout.read()}"
                            )
                        if self.server_process.stderr:
                            error_message += (
                                f" stderr: {self.server_process.stderr.read()}"
                            )
>                       raise RuntimeError(error_message)
E                       RuntimeError: Timed out while attempting to connect to ephemeral Prefect API server.

/home/vintecc/py310/lib/python3.10/site-packages/prefect/server/api/server.py:921: RuntimeError
---------------------------- Captured stdout setup -----------------------------
prefect home: /tmp/prefect-test-qdvfbrcv
---------------------------- Captured stderr setup -----------------------------
12:16:16.136 | INFO    | prefect - Starting temporary server on <http://127.0.0.1:8579>
See <https://docs.prefect.io/v3/concepts/server#how-to-guides> for more information on running a dedicated Prefect server.
12:16:18.410 | INFO    | uvicorn.error - Started server process [10365]
12:16:18.412 | INFO    | uvicorn.error - Waiting for application startup.
------------------------------ Captured log setup ------------------------------
INFO     prefect:server.py:881 Starting temporary server on <http://127.0.0.1:8579>
See <https://docs.prefect.io/v3/concepts/server#how-to-guides> for more information on running a dedicated Prefect server.
=========================== short test summary info ============================
ERROR python/fraio/fraio_orchestrator/tests/fraio_orchestrator/test_fraio_trigger_flow.py::test_prefect_settings_dump
========================= 1 error in 62.80s (0:01:02) ==========================
--- Logging error ---
Traceback (most recent call last):
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/logging/handlers.py", line 355, in emit
    self.console.print(message, soft_wrap=True)
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 1697, in print
    with self:
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 870, in __exit__
    self._exit_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 826, in _exit_buffer
    self._check_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 2038, in _check_buffer
    self._write_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 2107, in _write_buffer
    self.file.write(text)
ValueError: I/O operation on closed file.
Call stack:
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/server/api/server.py", line 960, in stop
    <http://subprocess_server_logger.info|subprocess_server_logger.info>(
  File "/usr/lib/python3.10/logging/__init__.py", line 1477, in info
    self._log(INFO, msg, args, **kwargs)
  File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log
    self.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle
    self.emit(record)
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/logging/handlers.py", line 361, in emit
    self.handleError(record)
Message: 'Stopping temporary server on <http://127.0.0.1:8579>'
Arguments: ()
m
Your message was too long, here's your message at the allowed limit: ok so let's try again where we're at
Copy code
os environ[ PREFECT_SERVER_LOGGING_LEVEL ] = INFO os environ[ PREFECT_UI_ENABLED ] = 0 # Optional force a known-free port and host mirroring your working CLI setup os environ[ PREFECT_SERVER_API_HOST ] = 127 0 0 1 os environ[ PREFECT_SERVER_API_PORT ] = 4201 # Strongly recommended isolate Prefect state/logs to a temp directory for tests import tempfile os environ[ PREFECT_HOME ] = tempfile mkdtemp(prefix= prefect-test- ) # Ensure we never inherit a Cloud URL os environ pop( PREFECT_API_URL None) @pytest fixture(scope= session autouse=True) def prefect_test_server() print(f prefect home {os environ['PREFECT_HOME']} ) with prefect_test_harness(server_startup_timeout=60) yield def test_prefect_settings_dump() s = get_current_settings() print( API URL s api_url) print( HOME s home) print( LOG TO CONSOLE s logging_to_console)
results in ```============================= test session starts ============================== platform linux -- Python 3 10 12 pytest-7 4 4 pluggy-1 6 0 rootdir /workspace configfile pytest ini plugins devtools-0 12 2 docker-3 2 3 anyio-4 11 0 asyncio-0 23 8 xdist-3 8 0 postgresql-7 0 2 dash-3 2 0 hydra-core-1 3 2 Faker-37 8 0 cov-6 2 1 asyncio mode=strict collected 1 item python/fraio/fraio_orchestrator/tests/fraio_orchestrator/test_fraio_trigger_flow py E [100%] ==================================== ERRORS ==================================== _________________ ERROR at setup of test_prefect_settings_dump _________________ @pytest fixture(scope= session autouse=True) def prefect_test_server() print(f prefect home {os environ['PREFECT_HOME']} ) > with prefect_test_harness(server_startup_timeout=60) python/fraio/fraio_orchestrator/tests/fraio_orchestrator/test_fraio_trigger_flow py 43 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib/python3 10/contextlib py 135 in enter return next(self gen) /home/vintecc/py310/lib/python3 10/site-packages/prefect/testing/utilities py 168 in prefect_test_harness test_server start( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <prefect server api server SubprocessASGIServer object at 0x7787051b7100> timeout = 60 def start(self timeout Optional[int] = None) -> None Start the server in a separate process Safe to call multiple times; only starts the server once Args timeout The maximum time to wait for the server to start if not self running if self port is None self port = self find_available_port() assert self port is not None Port must be provided or available help_message = ( f Starting temporary server on {self address}\nSee <https //docs prefect io/v3/concepts/server#how-to-guides> for more information on running a dedicated Prefect server ) <http //subprocess_server_logger info|subprocess_server_logger info>(help_message) try self running = True self server_process = self _run_uvicorn_command() atexit register(self stop) with httpx Client() as client response = None elapsed_time = 0 max_wait_time = ( timeout or PREFECT_SERVER_EPHEMERAL_STARTUP_TIMEOUT_SECONDS value() ) while elapsed_time < max_wait_time if self server_process poll() == 3 self port = self find_available_port() self server_process = self _run_uvicorn_command() continue try response = client get(f {self api_url}/health ) except httpx ConnectError pass else if response status_code == 200 break time sleep(0 1) elapsed_time += 0 1 if response response raise_for_status() if not response error_message
k
@Marvin ok, let's try again:
Copy code
============================= test session starts ==============================
platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.6.0
rootdir: /workspace
configfile: pytest.ini
plugins: devtools-0.12.2, docker-3.2.3, anyio-4.11.0, asyncio-0.23.8, xdist-3.8.0, postgresql-7.0.2, dash-3.2.0, hydra-core-1.3.2, Faker-37.8.0, cov-6.2.1
asyncio: mode=strict
collected 1 item

python/fraio/fraio_orchestrator/tests/fraio_orchestrator/test_fraio_trigger_flow.py E [100%]

==================================== ERRORS ====================================
_________________ ERROR at setup of test_prefect_settings_dump _________________

    @pytest.fixture(scope="session", autouse=True)
    def prefect_test_server():
        print(f"prefect home: {os.environ['PREFECT_HOME']}")
    
>       with prefect_test_harness(server_startup_timeout=60):

python/fraio/fraio_orchestrator/tests/fraio_orchestrator/test_fraio_trigger_flow.py:43: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python3.10/contextlib.py:135: in __enter__
    return next(self.gen)
/home/vintecc/py310/lib/python3.10/site-packages/prefect/testing/utilities.py:168: in prefect_test_harness
    test_server.start(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <prefect.server.api.server.SubprocessASGIServer object at 0x7787051b7100>
timeout = 60

    def start(self, timeout: Optional[int] = None) -> None:
    
>                       raise RuntimeError(error_message)
E                       RuntimeError: Timed out while attempting to connect to ephemeral Prefect API server.

/home/vintecc/py310/lib/python3.10/site-packages/prefect/server/api/server.py:921: RuntimeError
---------------------------- Captured stdout setup -----------------------------
prefect home: /tmp/prefect-test-qdvfbrcv
---------------------------- Captured stderr setup -----------------------------
12:16:16.136 | INFO    | prefect - Starting temporary server on <http://127.0.0.1:8579>
See <https://docs.prefect.io/v3/concepts/server#how-to-guides> for more information on running a dedicated Prefect server.
12:16:18.410 | INFO    | uvicorn.error - Started server process [10365]
12:16:18.412 | INFO    | uvicorn.error - Waiting for application startup.
------------------------------ Captured log setup ------------------------------
INFO     prefect:server.py:881 Starting temporary server on <http://127.0.0.1:8579>
See <https://docs.prefect.io/v3/concepts/server#how-to-guides> for more information on running a dedicated Prefect server.
=========================== short test summary info ============================
ERROR python/fraio/fraio_orchestrator/tests/fraio_orchestrator/test_fraio_trigger_flow.py::test_prefect_settings_dump
========================= 1 error in 62.80s (0:01:02) ==========================
--- Logging error ---
Traceback (most recent call last):
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/logging/handlers.py", line 355, in emit
    self.console.print(message, soft_wrap=True)
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 1697, in print
    with self:
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 870, in __exit__
    self._exit_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 826, in _exit_buffer
    self._check_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 2038, in _check_buffer
    self._write_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 2107, in _write_buffer
    self.file.write(text)
ValueError: I/O operation on closed file.
Call stack:
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/server/api/server.py", line 960, in stop
    <http://subprocess_server_logger.info|subprocess_server_logger.info>(
  File "/usr/lib/python3.10/logging/__init__.py", line 1477, in info
    self._log(INFO, msg, args, **kwargs)
  File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log
    self.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle
    self.emit(record)
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/logging/handlers.py", line 361, in emit
    self.handleError(record)
Message: 'Stopping temporary server on <http://127.0.0.1:8579>'
Arguments: ()
m
Your message was too long, here's your message at the allowed limit: ok let's try again ```============================= test session starts ============================== platform linux -- Python 3 10 12 pytest-7 4 4 pluggy-1 6 0 rootdir /workspace configfile pytest ini plugins devtools-0 12 2 docker-3 2 3 anyio-4 11 0 asyncio-0 23 8 xdist-3 8 0 postgresql-7 0 2 dash-3 2 0 hydra-core-1 3 2 Faker-37 8 0 cov-6 2 1 asyncio mode=strict collected 1 item python/fraio/fraio_orchestrator/tests/fraio_orchestrator/test_fraio_trigger_flow py E [100%] ==================================== ERRORS ==================================== _________________ ERROR at setup of test_prefect_settings_dump _________________ @pytest fixture(scope= session autouse=True) def prefect_test_server() print(f prefect home {os environ['PREFECT_HOME']} ) > with prefect_test_harness(server_startup_timeout=60) python/fraio/fraio_orchestrator/tests/fraio_orchestrator/test_fraio_trigger_flow py 43 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ /usr/lib/python3 10/contextlib py 135 in enter return next(self gen) /home/vintecc/py310/lib/python3 10/site-packages/prefect/testing/utilities py 168 in prefect_test_harness test_server start( _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = <prefect server api server SubprocessASGIServer object at 0x7787051b7100> timeout = 60 def start(self timeout Optional[int] = None) -> None > raise RuntimeError(error_message) E RuntimeError Timed out while attempting to connect to ephemeral Prefect API server /home/vintecc/py310/lib/python3 10/site-packages/prefect/server/api/server py 921 RuntimeError ---------------------------- Captured stdout setup ----------------------------- prefect home /tmp/prefect-test-qdvfbrcv ---------------------------- Captured stderr setup ----------------------------- 12 16 16 136 | INFO | prefect - Starting temporary server on <http //127 0 0 1 8579> See <https //docs prefect io/v3/concepts/server#how-to-guides> for more information on running a dedicated Prefect server 12 16 18 410 | INFO | uvicorn error - Started server process [10365] 12 16 18 412 | INFO | uvicorn error - Waiting for application startup ------------------------------ Captured log setup ------------------------------ INFO prefect server py 881 Starting temporary server on <http //127 0 0 1 8579> See <https //docs prefect io/v3/concepts/server#how-to-guides> for more information on running a dedicated Prefect server =========================== short test summary info ============================ ERROR python/fraio/fraio_orchestrator/tests/fraio_orchestrator/test_fraio_trigger_flow py test_prefect_settings_dump ========================= 1 error in 62 80s (0 01 02) ========================== --- Logging error --- Traceback (most recent call last) File /home/vintecc/py310/lib/python3 10/site-packages/prefect/logging/handlers py line 355 in emit self console print(message soft_wrap=True) File /home/vintecc/py310/lib/python3 10/site-packages/rich/console py line 1697 in print with self File /home/vintecc/py310/lib/python3 10/site-packages/rich/console py line 870 in exit self _exit_buffer() File /home/vintecc/py310/lib/python3 10/site-packages/rich/console py line 826 in _exit_buffer self _check_buffer() File /home/vintecc/py310/lib/python3 10/site-packages/rich/console py line 2038 in _check_buffer self _write_buffer() File /home/vintecc/py310/lib/python3 10/site-packages/rich/console py line 2107 in _write_buffer self file write(text) ValueError I/O operation on closed file Call stack File /home/vintecc/py310/lib/python3 10/site-packages/prefect/server/api/server py line 960 in stop <http //subprocess_server_logger info|subprocess_server_logger info>( File /usr/lib/python3 10/logging/__init__ py line 1477 in info self _log(INFO msg args **kwargs) File /usr/lib/python3 10/logging/__init__ py line 1624 in _log self handle(record) File /usr/lib/pyth
on3 10/logging/__init__ py line 1634 in handle self callHandlers(record) File /usr/lib/python3 10/logging/__init__ py line 1696 in callHandlers hdlr handle(record) File /usr/lib/python3 10/logging/__init__
k
@Marvin c'mon, I want to give you context. OK, so without context:
Copy code
os.environ["PREFECT_SERVER_LOGGING_LEVEL"] = "INFO"
os.environ["PREFECT_UI_ENABLED"] = "0"
# Optional: force a known-free port and host, mirroring your working CLI setup
os.environ["PREFECT_SERVER_API_HOST"] = "127.0.0.1"
os.environ["PREFECT_SERVER_API_PORT"] = "4201"
# Strongly recommended: isolate Prefect state/logs to a temp directory for tests
import tempfile

os.environ["PREFECT_HOME"] = tempfile.mkdtemp(prefix="prefect-test-")
# Ensure we never inherit a Cloud URL
os.environ.pop("PREFECT_API_URL", None)


@pytest.fixture(scope="session", autouse=True)
def prefect_test_server():
    print(f"prefect home: {os.environ['PREFECT_HOME']}")

    with prefect_test_harness(server_startup_timeout=60):
        yield


def test_prefect_settings_dump():
    s = get_current_settings()
    print("API URL:", s.api_url)
    print("HOME:", s.home)
    print("LOG TO CONSOLE:", s.logging_to_console)
result:
Copy code
ERROR python/fraio/fraio_orchestrator/tests/fraio_orchestrator/test_fraio_trigger_flow.py::test_prefect_settings_dump
========================= 1 error in 62.80s (0:01:02) ==========================
--- Logging error ---
Traceback (most recent call last):
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/logging/handlers.py", line 355, in emit
    self.console.print(message, soft_wrap=True)
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 1697, in print
    with self:
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 870, in __exit__
    self._exit_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 826, in _exit_buffer
    self._check_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 2038, in _check_buffer
    self._write_buffer()
  File "/home/vintecc/py310/lib/python3.10/site-packages/rich/console.py", line 2107, in _write_buffer
    self.file.write(text)
ValueError: I/O operation on closed file.
Call stack:
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/server/api/server.py", line 960, in stop
    <http://subprocess_server_logger.info|subprocess_server_logger.info>(
  File "/usr/lib/python3.10/logging/__init__.py", line 1477, in info
    self._log(INFO, msg, args, **kwargs)
  File "/usr/lib/python3.10/logging/__init__.py", line 1624, in _log
    self.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1634, in handle
    self.callHandlers(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
    hdlr.handle(record)
  File "/usr/lib/python3.10/logging/__init__.py", line 968, in handle
    self.emit(record)
  File "/home/vintecc/py310/lib/python3.10/site-packages/prefect/logging/handlers.py", line 361, in emit
    self.handleError(record)
Message: 'Stopping temporary server on <http://127.0.0.1:8579>'
Arguments: ()
Finished running tests!
m
✅ thought for 17.7 seconds
Thanks for the concrete snippet. In your latest run, the harness still times out because the parent process cannot reach the ephemeral server’s API URL even though the server subprocess starts. Two key fixes: 1) Disable proxies for localhost and do it before any Prefect import If any of HTTP_PROXY/HTTPS_PROXY/ALL_PROXY are set, httpx (used by Prefect) will route 127.0.0.1 through the proxy and the readiness check will hang. Add this at the very top of conftest.py (before importing Prefect or your code):
Copy code
# conftest.py (top)
import os, tempfile

# 1) Disable proxies for localhost
for k in ("HTTP_PROXY","HTTPS_PROXY","ALL_PROXY","http_proxy","https_proxy","all_proxy"):
    os.environ.pop(k, None)
os.environ["NO_PROXY"] = "127.0.0.1,localhost"
os.environ["no_proxy"] = "127.0.0.1,localhost"

# 2) Make server binding explicit and isolate state
os.environ["PREFECT_SERVER_API_HOST"] = "127.0.0.1"
os.environ["PREFECT_SERVER_API_PORT"] = "4201"
os.environ["PREFECT_UI_ENABLED"] = "0"
os.environ["PREFECT_HOME"] = tempfile.mkdtemp(prefix="prefect-test-")
os.environ.pop("PREFECT_API_URL", None)

# 3) Silence console logging to avoid Rich tearing down pytest’s stream
os.environ["PREFECT_LOGGING_TO_CONSOLE"] = "0"
os.environ["PREFECT_LOGGING_RICH_CONSOLE"] = "0"
os.environ["PREFECT_LOGGING_TO_FILE"] = "1"
os.environ["PREFECT_SERVER_LOGGING_LEVEL"] = "INFO"
2) Keep your harness fixture, just with a longer timeout
Copy code
import pytest
from prefect.testing.utilities import prefect_test_harness

@pytest.fixture(scope="session", autouse=True)
def prefect_test_server():
    print(f"prefect home: {os.environ['PREFECT_HOME']}")
    with prefect_test_harness(server_startup_timeout=120):
        yield
Smoke test (unchanged):
Copy code
def test_prefect_settings_dump():
    from prefect.settings import get_current_settings
    s = get_current_settings()
    print("API URL:", s.api_url)
    print("HOME:", s.home)
    print("LOG TO CONSOLE:", s.logging_to_console)
Why this fixes your case - The server subprocess is starting; the readiness probe and your client were likely trying to hit 127.0.0.1 via a proxy. Setting NO_PROXY and unsetting proxy envs makes the health check hit the local server directly. - Disabling console logging avoids the Rich “I/O operation on closed file” noise when pytest closes captured stdout. If it still times out after this exact setup, please share the last ~100 lines of the Prefect log in PREFECT_HOME/logs/prefect.log — that will show the precise reason the health check can’t complete. Related docs: - Test workflows: Prefect 3 test workflows - Testing utilities and fixtures: prefect.testing utilities and fixtures
k
@Marvin I can't find the 
PREFECT_HOME
-that env is not set
m
✅ thought for 19.3 seconds
That’s expected. 
PREFECT_HOME
doesn’t need to be set in the environment; Prefect computes a default (usually ~/.prefect on Linux/macOS, %USERPROFILE%\.prefect on Windows). So 
os.environ["PREFECT_HOME"]
may not exist unless you explicitly set it. How to see the actual home Prefect is using - From code:
Copy code
from prefect.settings import get_current_settings
s = get_current_settings()
print("Prefect home:", s.home)
- If you want file logs, enable them and see where they go:
Copy code
import os
from pathlib import Path
from prefect.settings import get_current_settings

os.environ["PREFECT_LOGGING_TO_FILE"] = "1"  # set before Prefect initializes

s = get_current_settings()
print("Prefect home:", s.home)
print("Log file:", Path(s.home) / "logs" / "prefect.log")
How to force a temp home for tests - Set it before any Prefect import (e.g., top of conftest.py):
Copy code
import os, tempfile
os.environ["PREFECT_HOME"] = tempfile.mkdtemp(prefix="prefect-test-")
Key point: use 
get_current_settings().home
to discover the path Prefect resolved, not 
os.environ
.
k
@Marvin ok, I just can't get it to work, can we do it step by step, not in test files, but from a python console?
m
✅ thought for 30.6 seconds
Absolutely — let’s do this step-by-step from a clean Python console. Paste the following exactly as-is into a standard python REPL (not IPython/Jupyter). This will: - Disable proxies for localhost - Silence console logging to avoid Rich/pytest stream issues - Pin host/port - Isolate Prefect state to a temp directory - Start the ephemeral server via prefect_test_harness - Verify connectivity with a health check
Copy code
import os, tempfile, anyio
from pathlib import Path

# 1) Disable proxies for localhost so health checks don’t route through a proxy
for k in ("HTTP_PROXY","HTTPS_PROXY","ALL_PROXY","http_proxy","https_proxy","all_proxy"):
    os.environ.pop(k, None)
os.environ["NO_PROXY"] = "127.0.0.1,localhost"
os.environ["no_proxy"] = "127.0.0.1,localhost"

# 2) Make server binding explicit and silence console logs
os.environ["PREFECT_SERVER_API_HOST"] = "127.0.0.1"
os.environ["PREFECT_SERVER_API_PORT"] = "4201"
os.environ["PREFECT_UI_ENABLED"] = "0"
os.environ["PREFECT_LOGGING_RICH_CONSOLE"] = "0"
os.environ["PREFECT_LOGGING_TO_CONSOLE"] = "0"
os.environ["PREFECT_LOGGING_TO_FILE"] = "1"
os.environ["PREFECT_SERVER_LOGGING_LEVEL"] = "DEBUG"
os.environ.pop("PREFECT_API_URL", None)

# 3) Isolate Prefect state/logs to a temp directory
os.environ["PREFECT_HOME"] = tempfile.mkdtemp(prefix="prefect-test-")
print("PREFECT_HOME:", os.environ["PREFECT_HOME"])

# 4) Import Prefect after env is set
from prefect.testing.utilities import prefect_test_harness
from prefect.settings import get_current_settings
from prefect.client.orchestration import get_client

print("Before harness: API URL ->", get_current_settings().api_url)

# 5) Start ephemeral server and verify health
with prefect_test_harness(server_startup_timeout=120):
    s = get_current_settings()
    print("Inside harness: API URL ->", s.api_url)
    print("Log file ->", Path(s.home) / "logs" / "prefect.log")

    async def ping():
        async with get_client() as c:
            resp = await c.read_healthcheck()
            print("Healthcheck OK ->", resp)

    anyio.run(ping)

print("Done.")
What to expect - “PREFECT_HOME:” prints a temp directory - “Inside harness: API URL -> http://127.0.0.1:4201/api” - “Healthcheck OK -> …” - If it still times out, open the printed log file path (PREFECT_HOME/logs/prefect.log) and share the last ~100 lines — that will show exactly why readiness didn’t complete. If you’re running this in IPython/Jupyter and see an “event loop is running” error, replace the last line with:
Copy code
import asyncio
asyncio.run(ping())
Once this works in the console, we can mirror the same env setup at the top of your conftest.py and the tests should pass.
3 Views
Open in Slack
PreviousNext
Powered by