<@ULVA73B9P> I want to improve our unit test infra...
# ask-marvin
a
@Marvin I want to improve our unit test infrastructure. We're already using the pytest fixture for the test harness. however what I see is that depending on where we run tests, we get different behaviour. If I run locally on my computer, and forget to set up the harness, prefect API methods hit the configured cloud account that I have in my local prefect config. If I run inside our dev docker environment the PREFECT_API_URL env variable takes preference and we hit our local dev prefect server. We mark all of our prefect tests with a pytest mark, but there's a risk that we test a code flow that hits prefect but forget to force it to use the test harness server. We don't want to start the test harness globally for the whole suite, because that's a startup penalty for running non-prefect tests. Is there a recommended way to cause a quick failure for a test if we forget to start the test harness? I was thinking of doing something like forcing PREFECT_API_URL to an invalid value in tests and making it not automatically start the test server in an ad-hoc way, and also never use
prefect config
-based configuration?
m
thought for 211.0 seconds
Great question — and you’re spot on to want a fast, consistent failure when the harness isn’t running. In Prefect 3.x, the cleanest pattern is: - Set safe, session-wide guardrails so any accidental API use fails immediately (no global harness startup, no cross-env surprises). - Opt-in to the harness per-test or per-module where needed. - Optionally enforce that your marked tests must be inside a harness. Below is a minimal setup you can drop into tests/conftest.py that achieves this. 1) Guardrails: force fast-fail unless a harness explicitly overrides - Disallow ephemeral server so Prefect won’t “helpfully” start one. - Set an invalid API URL so any stray client call errors out immediately. - Clear profile-based config influence in tests.
Copy code
# tests/conftest.py
import os
import pytest

@pytest.fixture(scope="session", autouse=True)
def _prefect_test_guardrails(monkeypatch):
    # Make accidental API use fail fast
    monkeypatch.setenv("PREFECT_SERVER_ALLOW_EPHEMERAL_MODE", "false")
    # Unroutable/invalid URL to guarantee immediate failure on network use
    monkeypatch.setenv("PREFECT_API_URL", "<http://127.0.0.1:0>")
    # Avoid picking up a local profile during tests
    monkeypatch.delenv("PREFECT_PROFILE", raising=False)
    yield
Why this works:
prefect.testing.utilities.prefect_test_harness()
uses
temporary_settings()
which takes precedence over env vars, so it will safely override the invalid URL inside tests that opt-in to the harness. Everywhere else, accidental API calls fail immediately without starting any server. 2) Opt-in harness fixture you can use per test
Copy code
# tests/conftest.py (continued)
import pytest
from prefect.testing.utilities import prefect_test_harness

@pytest.fixture
def prefect_harness():
    with prefect_test_harness():
        yield
Usage options: - Use it explicitly in a test:
Copy code
def test_my_flow(prefect_harness):
    ...
- Or enforce it via a marker you already use:
Copy code
import pytest

@pytest.mark.usefixtures("prefect_harness")
def test_flow_marked_prefect():
    ...
3) Optional: fail fast when a “prefect-marked” test forgot the harness If you already mark tests that might hit Prefect (e.g., @pytest.mark.prefect), add a small guard that fails those tests if the harness was not started.
Copy code
# tests/conftest.py (continued)
import pytest
from prefect.settings import PREFECT_API_URL

@pytest.fixture(autouse=True)
def _require_harness_for_prefect_mark(request):
    if request.node.get_closest_marker("prefect"):
        api_url = PREFECT_API_URL.value()
        if not api_url or api_url == "<http://127.0.0.1:0>":
            pytest.fail(
                "This test is marked @pytest.mark.prefect but no test harness/API is running. "
                "Use `with prefect_test_harness():` or `@pytest.mark.usefixtures('prefect_harness')`."
            )
Notes and tips - Keep Prefect client creation inside tests/functions, not at import time. Pytest fixtures apply after import; if you must guard even imports, set the same env vars in your test runner/CI command (e.g., export PREFECT_SERVER_ALLOW_EPHEMERAL_MODE=false and PREFECT_API_URL=http://127.0.0.1:0 before pytest runs). - This approach avoids starting a server globally, keeps non-Prefect tests fast, and guarantees quick failure anywhere you forgot the harness. Docs and source links - How to test workflows: How to test workflows - Testing utilities (includes prefect_test_harness): API ref: prefect.testing.utilities - Testing fixtures: API ref: prefect.testing.fixtures - Source: prefect_test_harness implementation utilities.py
- Source: fixtures (hosted_api_server, use_hosted_api_server, disable_hosted_api_server, enable_ephemeral_server) fixtures.py If you want, I can help tailor the guard to a different marker name (or invert it to allow-list specific modules). Are you on Prefect 3.x? The above assumes 3.x semantics.
a
great. fwiw if anyone else is following this, this is exactly the same pattern that codex gave me after feeding it the same request