https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> In Prefect 3.7.0 when running `prefec...
# ask-marvin
r
@Marvin In Prefect 3.7.0 when running 
prefect deploy
from command line for a flow, it errors out because it cannot load a Block of a specific name. This seems to be because one of the utility functions imported into the flow loads that named Block. But it only executes the loading when the function is actually called, and we're only at prefect deploy and not trying to run the flow, i.e. just get it deployed. How to work around this?
m
✅ thought for 175.0 seconds
Yes — 
prefect deploy
does import and inspect your flow module, so anything that gets executed during import / function definition time can fail deployment, even though you’re not running the flow yet. In practice, this usually happens because one of these is true: - there is 
Block.load("name")
in top-level module code - there is 
Block.load("name")
in a default argument - an imported helper module does work at import time and indirectly triggers the block load What to do instead The workaround is to make block loading lazy so it only happens at runtime, inside the flow/task/function body. Good pattern
Copy code
from prefect import flow
from prefect.blocks.core import Block

def get_my_block():
    return Block.load("my-block-name")

@flow
def my_flow():
    block = get_my_block()
    # use block here
Bad pattern
Copy code
from prefect import flow
from prefect.blocks.core import Block

MY_BLOCK = Block.load("my-block-name")  # runs during import, so deploy can fail

@flow
def my_flow():
    ...
Also avoid this:
Copy code
@flow
def my_flow(cfg=Block.load("my-block-name")):
    ...
because default arguments are evaluated when the function is defined, which happens during module import. Recommended fixes 1. Move 
Block.load(...)
into the flow body, task body, or a helper function called from there 2. Remove module-level globals that load blocks 3. Replace default-argument loads with 
None
+ runtime resolution Example:
Copy code
from prefect import flow
from prefect.blocks.system import Secret

def get_api_key():
    return Secret.load("my-secret").get()

@flow
def my_flow(api_key=None):
    if api_key is None:
        api_key = get_api_key()
    print(api_key)
If your helper module currently does this:
Copy code
from prefect.blocks.system import Secret
SECRET = Secret.load("my-secret")
change it to:
Copy code
from prefect.blocks.system import Secret

def get_secret():
    return Secret.load("my-secret")
Then call 
get_secret()
only from inside the running flow/task. Why this happens 
prefect deploy
needs to load your flow object to inspect it and build the deployment, so Prefect imports the module referenced by your entrypoint. That import executes Python code normally. 
Block.load()
talks to the Prefect API immediately, so if the named block isn’t available in that environment, deploy fails before registration. If you want, I can help you pinpoint the exact pattern in your code if you paste the flow file and the helper that imports/loads the block.
r
@Marvin I don't think I have anything at top level module or as a default argument. Is there a way to narrow down where it is trying to load the Block at prefect deploy time? e.g. verbosity or some other way to find the line of code that is causing the problem?
m
✅ thought for 107.0 seconds
Yes — there are a few good ways to narrow this down. The short version: 
prefect deploy
does not have a special “show me the offending source line” mode, but you can usually find it by forcing a full Python traceback and/or instrumenting 
Block.load()
. First thing to try: run with debug logging and full traceback
Copy code
PREFECT_LOGGING_LEVEL=DEBUG prefect deploy ...
Also capture stderr so you can inspect the traceback carefully:
Copy code
PREFECT_LOGGING_LEVEL=DEBUG prefect deploy ... 2>&1 | tee deploy.log
If Prefect is swallowing some detail, the more reliable approach is to reproduce the same import outside the CLI. Best debugging trick: import the flow module directly with Python Since deploy fails while importing/introspecting your flow, try importing the exact entrypoint yourself:
Copy code
python -c "from path.to.flow_file import my_flow"
or if it’s a file path rather than a package import:
Copy code
python flow_file.py
That often gives a cleaner native Python traceback showing the exact file and line where the block load is triggered. If the flow is normally deployed from an entrypoint like:
Copy code
prefect deploy path/to/flow_file.py:my_flow
then a very good reproduction is:
Copy code
python -c "import importlib.util; spec = importlib.util.spec_from_file_location('m', 'path/to/flow_file.py'); m = importlib.util.module_from_spec(spec); spec.loader.exec_module(m)"
That mimics the “load this module” behavior more closely than a normal package import. Most useful way to pinpoint it exactly: monkey-patch 
Block.load()
 You can temporarily add this at the very top of your flow file, before other imports that may trigger the issue:
Copy code
import traceback
from prefect.blocks.core import Block

_original_load = Block.load

@classmethod
def _debug_load(cls, *args, **kwargs):
    print("\n=== Block.load called ===")
    print(f"class={cls.__name__}, args={args}, kwargs={kwargs}")
    traceback.print_stack(limit=20)
    return _original_load.__get__(None, cls)(*args, **kwargs)

Block.load = _debug_load
Then run 
prefect deploy
again. When the block load happens, you should see the stack showing which import/function/class/property triggered it. If your code uses async block loading too, you may want to patch `aload`/`aload_from_ref` as well, but 
Block.load()
is the main one to start with. Also search for less obvious trigger points Even if you don’t have obvious top-level code or default args, these can still trigger during import: - class attributes - dataclass field defaults / 
default_factory
- pydantic model field defaults - decorators - properties accessed while building objects - module-level singleton/config initialization in imported helper modules - code inside 
__init__.py
- imported constants that are computed from a block I’d grep broadly for 
load(
and block classes, not just obvious top-level lines:
Copy code
grep -R "Block.load\|Secret.load\|String.load\|JSON.load\|Credentials.load" .
and also look for your specific block name:
Copy code
grep -R "your-block-name" .
Another strong clue: check 
prefect.yaml
too Sometimes the load is not coming from Python import at all, but from deployment config resolution. For example if 
prefect.yaml
contains block references, Prefect may resolve them during deploy. Search for things like: - 
prefect.blocks
- 
$ref
- block document references Example:
Copy code
grep -R "prefect.blocks\|\$ref" prefect.yaml .
Practical debugging order I’d use 1. Run with debug logs:
Copy code
PREFECT_LOGGING_LEVEL=DEBUG prefect deploy ... 2>&1 | tee deploy.log
2. Reproduce with plain Python import of the flow module 3. If still unclear, monkey-patch 
Block.load()
to print a stack trace 4. Search helper modules and 
prefect.yaml
for block references One important note If the failing load is happening from an imported utility module, the line may be nowhere near your 
@flow
definition. It may be in:
- that utility module’s import-time initialization - a class body - a config object instantiated during import So the stack trace from the monkey-patch is usually the fastest way to find the true source. If you want, paste: - your 
prefect deploy
command - the full error traceback - the flow entrypoint - and the helper module import and I can help you identify the likely trigger point.
r
@Marvin I've tried the money patch as suggested. The flow now has only the minimal possible line of code. So it's your example above from "top of your flow", then the offending import statement, followed by the flow itself which takes no parameters and only returns a None, i.e. simple. No extra debugging or logger or whatever. Still the same cryptic unable to find block named ... What else can I try, seems the monkey patch wasnt effective. For additional context, I am using uv so presumably it will be looking in the .venv directory when doing the import
m
✅ thought for 206.6 seconds
Yep — that explains why the monkey patch didn’t help. Key point: the lookup during 
prefect deploy
may not be going through 
Block.load()
at all. In Prefect 3.x, deploy-time block resolution often happens through deployment config templating and direct client API calls like: - 
read_block_document_by_name(...)
- 
read_block_document(...)
So patching 
Block.load()
in your flow file won’t catch those. What this means for your case If your flow file is now minimal and the error still happens, the most likely cause is not the flow import itself anymore. It’s probably one of these: 1. a block reference in 
prefect.yaml
2. a block reference in a build/push/pull step config 3. a work pool / job variable reference being resolved during deploy 4. some deployment config generated from the CLI First thing I would check Search your repo for Prefect block templating syntax:
Copy code
grep -R "prefect.blocks\|\$ref" .
Especially inspect 
prefect.yaml
for things like:
Copy code
{{ prefect.blocks.secret.my-secret }}
{{ prefect.blocks.some-block-type.some-block-name.some_field }}
or:
Copy code
$ref:
  block_document_id: ...
Also check any deployment steps under 
build:
, 
push:
, or 
pull:
. Why this is likely Prefect resolves block references in deployment config before applying the deployment, and that path bypasses 
Block.load()
entirely. Best next debugging step: patch the client lookup, not Block.load() Try this in a separate script rather than inside the flow file. Create `debug_deploy.py`:
Copy code
import traceback

from prefect.client.orchestration._blocks_types.client import SyncBlocksTypeClient
from prefect.client.orchestration._blocks_documents.client import SyncBlocksDocumentClient

orig_by_name = SyncBlocksTypeClient.read_block_document_by_name
orig_by_id = SyncBlocksDocumentClient.read_block_document

def traced_by_name(self, name, block_type_slug, *args, **kwargs):
    print(f"\n=== read_block_document_by_name called ===")
    print(f"block_type_slug={block_type_slug}, name={name}")
    traceback.print_stack(limit=25)
    return orig_by_name(self, name, block_type_slug, *args, **kwargs)

def traced_by_id(self, block_document_id, *args, **kwargs):
    print(f"\n=== read_block_document called ===")
    print(f"block_document_id={block_document_id}")
    traceback.print_stack(limit=25)
    return orig_by_id(self, block_document_id, *args, **kwargs)

SyncBlocksTypeClient.read_block_document_by_name = traced_by_name
SyncBlocksDocumentClient.read_block_document = traced_by_id

from prefect.cli import app

if __name__ == "__main__":
    app()
Then run deploy through that script with `uv`:
Copy code
uv run python debug_deploy.py deploy ...
This should print the stack trace for the exact block lookup path. Why use a wrapper script? Because if Prefect is resolving block references before your flow module is imported, code placed in the flow file is too late. A wrapper script that patches Prefect before the CLI starts is much more reliable. If you're using 
uv
 Yes, 
uv
will run inside the project environment, typically 
.venv
, but that itself is not the core issue. The important thing is that the wrapper script runs in the same environment as the 
prefect
command, which 
uv run python debug_deploy.py ...
does. Also useful: isolate whether the problem is config vs import Try both of these: 1. Deploy without using 
prefect.yaml
, if possible, by passing the entrypoint directly and simplifying options 2. Temporarily move 
prefect.yaml
aside and run a minimal deploy For example:
Copy code
mv prefect.yaml prefect.yaml.bak
uv run prefect deploy path/to/flow.py:my_flow
If the error disappears, the culprit is almost certainly in 
prefect.yaml
or deployment step configuration. Very likely diagnosis from your description Since: - your flow file is now minimal - patching 
Block.load()
did nothing - the same missing block error still occurs
the block lookup is probably coming from deployment configuration resolution, not Python flow import. If you want, paste your 
prefect.yaml
and I can point out the exact places where Prefect 3.7.0 would try to resolve blocks during 
prefect deploy
.
r
@Marvin I made a simple prefect.yaml and that didn't fix the problem. I'll try your debug_deploy.py next
m
✅ thought for 8.0 seconds
That’s a good next step. If a stripped-down 
prefect.yaml
still doesn’t fix it, that shifts suspicion back toward: - flow/module import side effects in some imported dependency - or a deploy-time code path outside your flow file but still inside the Python environment A couple of tips before you run `debug_deploy.py`: Run it through 
uv
in the same environment
Copy code
uv run python debug_deploy.py deploy path/to/flow.py:my_flow
or if you normally pass 
--name
etc, include the same args. If the import path for the client classes differs Depending on the exact Prefect build, those internal client import paths can move around a bit. If the script errors on import, tell me the exact import error and I can adjust it. What success looks like If the block lookup is happening through Prefect’s client path, you should see output like:
Copy code
=== read_block_document_by_name called ===
block_type_slug=...
name=...
...
with a stack trace. That trace is the important part. If 
debug_deploy.py
still shows nothing Then I’d try one of these two escalation steps: 1. Patch lower, at the HTTP client/request layer, so every API request is shown 2. Patch earlier, before any Prefect CLI imports, in case the lookup occurs extremely early A simple “show every Prefect API request” wrapper can be very effective if needed. When you try 
debug_deploy.py
, send me either: - the traceback/output it prints, or - any import error it hits and I’ll help you refine it.
r
@Marvin I have prefect installed in my virtual env, but when I try to run your debug_deploy.py it cannot import the prefect.client.orchastration modules
m
✅ thought for 12.4 seconds
Ah — thanks, that’s useful. I likely gave you an internal import path that doesn’t exist in your exact installed build. Let’s avoid brittle internal imports and debug this in a more robust way. Best fallback: patch the generic client request method This catches the actual API call regardless of which higher-level helper is using it. Try this 
debug_deploy.py
instead:
Copy code
import traceback

from prefect.client.orchestration.base import BaseClient

orig_request = BaseClient.request

def traced_request(self, method, path, *args, **kwargs):
    if "block" in str(path):
        print("\n=== Prefect API request involving blocks ===")
        print(f"method={method}")
        print(f"path={path}")
        print(f"args={args}")
        print(f"kwargs={kwargs}")
        traceback.print_stack(limit=25)
    return orig_request(self, method, path, *args, **kwargs)

BaseClient.request = traced_request

from prefect.cli import app

if __name__ == "__main__":
    app()
Then run:
Copy code
uv run python debug_deploy.py deploy ...
If that import also fails, then let’s confirm what exists in your installed Prefect. Quick inspection script Run this:
Copy code
uv run python -c "import prefect.client as c; import inspect; print(c); print(dir(c))"
and also:
Copy code
uv run python -c "import prefect; print(prefect.__version__)"
If 
prefect.client.orchestration.base
doesn’t exist, we can discover the right path by introspection. You can also list available submodules with:
Copy code
uv run python -c "import prefect.client.orchestration as o; print(dir(o))"
Even more robust alternative: patch HTTPX directly Prefect’s API client ultimately uses HTTP requests. If the block lookup is happening via the API, we can intercept those without depending on Prefect internals at all. Try this version:
Copy code
import traceback
import httpx

orig_send = httpx.Client.send
orig_async_send = httpx.AsyncClient.send

def traced_send(self, request, *args, **kwargs):
    url = str(request.url)
    if "block" in url:
        print("\n=== HTTPX request involving blocks ===")
        print(f"{request.method} {url}")
        traceback.print_stack(limit=25)
    return orig_send(self, request, *args, **kwargs)

async def traced_async_send(self, request, *args, **kwargs):
    url = str(request.url)
    if "block" in url:
        print("\n=== HTTPX async request involving blocks ===")
        print(f"{request.method} {url}")
        traceback.print_stack(limit=25)
    return await orig_async_send(self, request, *args, **kwargs)

httpx.Client.send = traced_send
httpx.AsyncClient.send = traced_async_send

from prefect.cli import app

if __name__ == "__main__":
    app()
Run it the same way:
Copy code
uv run python debug_deploy.py deploy ...
This should show you: - the exact API endpoint being hit - the stack trace leading to it One more likely issue: typo check You wrote 
orchastration
in your message. If that same spelling was in the script, that alone would cause the import failure. The correct spelling is:
Copy code
prefect.client.orchestration
My recommendation Use the 
httpx
patch version next — it’s the least fragile. If you paste the output of:
Copy code
uv run python -c "import prefect; print(prefect.__version__)"
and the result of the 
httpx
debug run, I can help interpret exactly where the lookup is coming from.
r
@Marvin I'm using your httpx version. It does run and I do get a long stack trace. I also can see the GET: https call to try and load the Block. But no where in the stack trace can I see where in my code or yamls or anything to see what is originating that GET
m
✅ thought for 15.9 seconds
That’s actually a very strong clue. If the HTTPX trace shows the 
GET
for the block, but the stack trace only shows Prefect internals and nothing from your repo, then the lookup is likely being triggered by Prefect’s deployment loading/introspection process itself, not by a direct 
Block.load()
call in your code. At this point, the most useful thing is to find which object/value Prefect is resolving right before that GET. The raw HTTP layer is too low-level. Next step: patch the templating/block-resolution layer and print the actual template data being resolved Try this `debug_deploy.py`:
Copy code
import traceback
import pprint

from prefect.utilities import templating

orig_resolve = templating.resolve_block_document_references

async def traced_resolve(template, client=None, value_transformer=None):
    print("\n=== resolve_block_document_references called ===")
    print("Template type:", type(template))
    try:
        pprint.pp(template, width=120)
    except Exception as exc:
        print(f"<<could not pprint template: {exc}>>")
    traceback.print_stack(limit=20)
    return await orig_resolve(template, client=client, value_transformer=value_transformer)

templating.resolve_block_document_references = traced_resolve

from prefect.cli import app

if __name__ == "__main__":
    app()
Then run:
Copy code
uv run python debug_deploy.py deploy ...
What this tells us This should show the actual deployment config / step input / parameter structure being scanned for block references. If the block name appears in that printed object, we’ll know what Prefect thinks it is resolving. If that still isn’t enough, patch the placeholder finder too:
Copy code
import traceback
from prefect.utilities import templating

orig_find = templating.find_placeholders

def traced_find(template):
    result = orig_find(template)
    blockish = [r for r in result if getattr(r, "type", None) and "BLOCK" in str(r.type)]
    if blockish:
        print("\n=== find_placeholders found block placeholders ===")
        print("template:", template)
        print("placeholders:", blockish)
        traceback.print_stack(limit=20)
    return result

templating.find_placeholders = traced_find
You can combine this with the previous patch. Why this is better than the HTTP trace The HTTP trace only tells you “Prefect is now looking up a block.” The templating trace can tell you “Prefect is looking up this block because it found this exact string/value/object.” Another high-value check: print the deploy config before resolution There may be an auto-generated deploy config coming from CLI defaults, work pool settings, or project metadata. If you want, we can also patch the deploy runner to print the full 
deploy_config
before block resolution. A useful wrapper is:
Copy code
import pprint
from prefect.cli.deploy import _core

orig_run = _core._run_single_deploy

async def traced_run_single_deploy(*args, **kwargs):
    deploy_config = kwargs.get("deploy_config")
    print("\n=== deploy_config before deploy ===")
    pprint.pp(deploy_config, width=120)
    return await orig_run(*args, **kwargs)

_core._run_single_deploy = traced_run_single_deploy
If the block name already appears there, then we know it is coming from deploy configuration, not your flow code. Most likely scenarios now Given what you’ve described, I’d rank them like this: 1. A block placeholder exists in the effective deploy config even if not in your simplified 
prefect.yaml
, it may be getting merged from another source 2. An imported module or deployment helper is generating config dynamically and returning a value that includes a block reference string 3. A worker/work-pool/job variable default is being pulled in and that contains a block reference What would help fastest now Please run the 
resolve_block_document_references
patch and paste: - the printed template object just before the failing GET
- or at least the portion containing the block name That should let me tell you exactly what Prefect is resolving and from where.
r
@Marvin Now I get the template output. I still have the error and the block name only appears in the error. i.e. still no insight. I need to now try your patch the placeholder finder too and the deploy_config. I need to get lunch, will you retain your context when I get back in, say 15 minutes?
m
✅ thought for 5.5 seconds
Yes — within this same Slack thread, I’ll retain the context of what we’ve already gone through. So when you come back, just continue here with: - the output from the 
find_placeholders
patch and/or - the printed 
deploy_config
- or even just the relevant excerpt around the failing block name Then I can pick up from there without you having to re-explain everything. A good next combo to run after lunch is: 1. patch 
find_placeholders
2. patch 
_run_single_deploy
to print 
deploy_config
3. keep the HTTPX patch only if needed Enjoy lunch — we can continue when you’re back.
r
@Marvin For the find_placeholders you told me to "You can combine this with the previous patch." There's so much stuff to this point, I'm not sure what you mean or how to do it. Can you show me explicitly?
m
✅ thought for 22.1 seconds
Yes — here’s an explicit single 
debug_deploy.py
that combines the two useful patches: - prints 
deploy_config
before deploy runs - prints whenever 
find_placeholders
finds block placeholders - prints whenever 
resolve_block_document_references
is called - optionally keeps the HTTPX block-request trace too Copy this whole file as-is: ``` import pprint import traceback import httpx from prefect.utilities import templating from prefect.cli.deploy import _core from prefect.cli import app # ----------------------------------------------------------------------------- # 1) Print deploy_config before Prefect starts resolving things # ----------------------------------------------------------------------------- orig_run_single_deploy = _core._run_single_deploy async def traced_run_single_deploy(*args, **kwargs): deploy_config = kwargs.get("deploy_config") print("\n================ DEPLOY CONFIG BEFORE _run_single_deploy ================") try: pprint.pp(deploy_config, width=120) except Exception as exc: print(f"<<could not pprint deploy_config: {exc}>>") print("========================================================================\n") return await orig_run_single_deploy(*args, **kwargs) _core._run_single_deploy = traced_run_single_deploy # ----------------------------------------------------------------------------- # 2) Print whenever Prefect scans a value and finds block placeholders # ----------------------------------------------------------------------------- orig_find_placeholders = templating.find_placeholders def traced_find_placeholders(template): result = orig_find_placeholders(template) blockish = [] for item in result: item_type = getattr(item, "type", None) if item_type and "BLOCK" in str(item_type): blockish.append(item) if blockish: print("\n================ BLOCK PLACEHOLDERS FOUND ================") print("Template being scanned:") try: pprint.pp(template, width=120) except Exception as exc: print(f"<<could not pprint template: {exc}>>") print("\nPlaceholders found:") for item in blockish: print(f" - {item}") print("\nStack trace:") traceback.print_stack(limit=20) print("==========================================================\n") return result templating.find_placeholders = traced_find_placeholders # ----------------------------------------------------------------------------- # 3) Print whenever Prefect tries to resolve block document references # ----------------------------------------------------------------------------- orig_resolve_block_document_references = templating.resolve_block_document_references async def traced_resolve_block_document_references(template, client=None, value_transformer=None): print("\n================ resolve_block_document_references CALLED ================") print(f"Template type: {type(template)}") try: pprint.pp(template, width=120) except Exception as exc: print(f"<<could not pprint template: {exc}>>") print("\nStack trace:") traceback.print_stack(limit=20) print("==========================================================================\n") return await orig_resolve_block_document_references( template, client=client, value_transformer=value_transformer, ) templating.resolve_block_document_references = traced_resolve_block_document_references # ----------------------------------------------------------------------------- # 4) OPTIONAL: Print HTTP requests involving blocks # ----------------------------------------------------------------------------- orig_httpx_client_send = httpx.Client.send orig_httpx_async_client_send = httpx.AsyncClient.send def traced_httpx_client_send(self, request, *args, **kwargs): url = str(request.url) if "block" in url:
print("\n================ HTTPX BLOCK REQUEST (sync) ================") print(f"{request.method} {url}") print("\nStack trace:") traceback.print_stack(limit=20) print("============================================================\n") return orig_httpx_client_send(self, request, *args, **kwargs) async def traced_httpx_async_client_send(self, request, *args, **kwargs): url = str(request.url) if "block" in url: print("\n================ HTTPX BLOCK REQUEST (async) ================") print(f"{request.method} {url}") print("\nStack trace:") traceback.print_stack(limit=20) print("=============================================================\n") return await orig_httpx_async_client_send(self, request, *args, **kwargs) httpx.Client.send = traced_httpx_client_send httpx.AsyncClient.send = traced_httpx_async_client_send # ----------------------------------------------------------------------------- # Entry point: run the Prefect CLI through this patched process # ----------------------------------------------------------------------------- if name == "__main__": app()
Copy code
Then run it with the same environment you normally use:
uv run python debug_deploy.py deploy ...
Copy code
For example:
uv run python debug_deploy.py deploy path/to/flow.py:my_flow
Copy code
or with your normal flags:
uv run python debug_deploy.py deploy path/to/flow.py:my_flow --name my-deployment
Copy code
*What to look for*

The most useful outputs will be:

1. *`DEPLOY CONFIG BEFORE _run_single_deploy`*  
   Check whether the missing block name appears anywhere there.

2. *`BLOCK PLACEHOLDERS FOUND`*  
   If this prints, look at:
   - `Template being scanned`
   - `Placeholders found`

3. *`resolve_block_document_references CALLED`*  
   If this prints many times, inspect the one right before the error.

4. *`HTTPX BLOCK REQUEST`*  
   Confirms the actual outgoing block lookup.

*If you want less noise*
If this is too noisy, we can simplify it to *only print when the block name appears*. If you know the exact missing block name, replace `MISSING_BLOCK_NAME` below:
TARGET = "MISSING_BLOCK_NAME"
Copy code
and then guard the print calls with checks like:
if TARGET in str(template): ... ``` If you want, I can give you a second, filtered version of 
debug_deploy.py
where you paste the block name once and it only prints relevant events. That may be easier to read.
r
@Marvin This is a hot mess. I ran your script and I still don't have any insight. I get 100s of lines of output. none of those looks like my code/repo, it's all prefect or other libraries. The final bit of output before it errors has: resolve_block_document_references. And it's a Template type: <class 'dict'>. But specific block name that it cannot load only appears on the final output line in a red error message, i.e. the same error we started this thread with. Can't you help me?
m
✅ thought for 21.0 seconds
Yes — let’s simplify and get practical. At this point, the big debugging wrappers are creating noise without isolating the cause. The key clue is: - the failing block name does not appear in the printed templates/config you inspected - the last visible thing is 
resolve_block_document_references(...)
- the error is still “cannot load block named ...” That strongly suggests one of two things: 1. the lookup is happening while importing your flow module or one of its imports, but not through 
Block.load
2. or the error message is being raised from a nested object/value that your prints are not surfacing cleanly So instead of tracing everything, let’s isolate the import chain directly. What I want you to do next Create a tiny script whose only job is to import your flow module and print the import chain. Use this exact script and replace the file path with your actual flow file.
Copy code
import builtins
import runpy
import traceback

TARGET_FILE = "path/to/your/flow_file.py"

orig_import = builtins.__import__

def traced_import(name, globals=None, locals=None, fromlist=(), level=0):
    print(f"IMPORT: name={name}, fromlist={fromlist}, level={level}")
    return orig_import(name, globals, locals, fromlist, level)

builtins.__import__ = traced_import

try:
    runpy.run_path(TARGET_FILE, run_name="__main__")
except Exception:
    print("\n=== EXCEPTION DURING IMPORT ===")
    traceback.print_exc()
finally:
    builtins.__import__ = orig_import
Run it with:
Copy code
uv run python debug_imports.py
Why this is better This removes 
prefect deploy
entirely and asks one narrow question: Can Python import the flow file cleanly? If it fails the same way, then the issue is definitely in: - the flow file itself - or one of the modules it imports If it doesn’t fail, then the problem is more specific to Prefect deploy-time inspection. Even better: binary search the imports Since you mentioned there is an “offending import statement”, do this manually: 1. comment out all non-Prefect imports from the flow file 2. run:
Copy code
uv run prefect deploy path/to/flow.py:my_flow
3. if it works, add imports back one at a time until it fails This is the fastest way to identify which module triggers the block lookup. Once you find the offending import, do the same process inside that module: - comment out its imports / initialization - add pieces back one by one I know this is manual, but it is often much faster than trying to decode Prefect’s internal stack traces. Very important: inspect package 
__init__.py
files A common hidden source is not the module you imported directly, but a package 
__init__.py
that runs code like: - config setup - singleton creation - secret loading - block lookup - environment bootstrap So if you import something like:
Copy code
from mypackage.utils import helper
also inspect: - 
mypackage/__init__.py
- 
mypackage/utils/__init__.py
Try importing the “offending import” by itself If you already suspect one import, test it directly:
Copy code
uv run python -c "from mypackage.some_module import some_symbol"
If that alone throws the missing block error, then you’ve proven it is outside Prefect deploy and just a normal Python import side effect. Most likely root cause now Given everything you’ve found, my strongest guess is: - the block lookup is triggered by a normal Python import side effect in some dependency/module - not by your flow body - and not by 
prefect.yaml
That would also explain why the block name only appears at the final error and not in the deploy config/template prints. Concrete plan from here I’d do these in order: 1. Run 
uv run python -c "import your_flow_module"
or the 
runpy
script above 2. Comment out all nonessential imports in the flow file 3. Add them back one by one 4. When one import causes the failure, test that import directly outside Prefect If you want, paste: - the exact import line you suspect
- the structure of that imported module - and whether 
uv run python -c "from ... import ..."
reproduces the error and I’ll help you narrow the likely side effect very specifically.
r
@Marvin Ok, thank you, this time it gave the insight we've been looking for! the problem was in the dunder init dunder.py. sigh. oh dodgy code written by a "Creator" years ago, good riddance. Hum a 1970s uplifting rock anthem to yourself to celebrate. And following that do "Another One Bites the Dust" for the nuking of that legacy init
m
✅ thought for 10.9 seconds
Nice — glad you found it. 
__init__.py
is exactly the kind of place that hides this sort of import-time side effect. Very believable, very annoying, and very satisfying to delete. Victory sequence engaged: - first: vague but uplifting 1970s arena-rock humming - then immediately: dum dum dum... another one bites the dust Sounds like the real fix was not a Prefect deploy issue so much as legacy Python import behavior masquerading as a Prefect issue. For future-proofing, the safe rule is: - keep 
__init__.py
files as close to empty as possible - avoid any config loading, block loading, network calls, or object construction at import time - especially avoid side effects in package-level exports If you want, I can help you turn what you found into a short “import hygiene” checklist for your repo so this doesn’t happen again.
r
@Marvin Using self hosted Prefect 3.7.0 in kubernetes, the flow run page for a given flow is slow to update the number of tasks that it has run and also the nice graphic of the flow run. It can take quite a bit of time, i.e. more than 10 mins and sometimes longer, to update. Looking to the kubernetes pods and logs and resource usage it all looks normal.
m
✅ thought for 221.9 seconds
Yes — there are a few plausible causes here, and one especially important behavior in Prefect 3.7.0: the flow run graph does not continuously poll the backend the same way some other run details do. So if the graph and task count/visual summary lag by 10+ minutes, that can be a mix of: - slow backend query generation for graph/task-run aggregates - stale UI refresh behavior - slow database response for graph/task-run endpoints - self-hosted database/config bottlenecks, even when pods “look normal” Most likely things to check first 1. What database are you using? If self-hosted Prefect is on SQLite, that is the first suspect. SQLite is not a good fit for production/self-hosted Kubernetes at this scale. PostgreSQL is strongly preferred. 2. How large are these flow runs? If these are runs with hundreds or thousands of task runs, the graph endpoint gets much more expensive. 3. Is it just the graph, or also the underlying API endpoints? The UI is only as fast as: - flow run details endpoint - task run count endpoint - graph endpoint If those are slow, the UI will lag even if pod CPU/memory looks fine. Important behavior in Prefect 3.7.0 From the code path research: - task counts are refreshed periodically - flow run details refresh more often while active - but the graph visualization itself is not aggressively polled So part of what you’re seeing may be UI behavior, but 10+ minutes is longer than I’d expect from UI polling alone. That suggests backend latency too. Most likely backend bottlenecks For self-hosted Kubernetes, the common ones are: - SQLite backend - Postgres underprovisioned / poor indexes / low connection pool - large task graphs causing expensive graph queries - background maintenance/vacuum causing lock contention - network latency between server and database - too-small API/database connection pool What to inspect next 1. Confirm database type If you are using SQLite, I would treat that as the primary issue and move to PostgreSQL. 2. Measure the actual API latency from the browser Open browser dev tools on the flow run page and watch requests for endpoints like: - 
/flow_runs/<id>/graph-v2
- task run count-related endpoints - flow run detail endpoints What matters: - do those requests take seconds/minutes? - or do they return fast and the UI just doesn’t repaint until later? 3. Check Prefect server DB settings The default SQLAlchemy pool settings can be too small for production-like use. Common knobs to review:
Copy code
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW
If you’re on Postgres and these are still defaults, increasing them may help. 4. Check flow run size Roughly how many tasks are in runs where this happens? - <100 tasks - hundreds - thousands+ The graph query cost rises significantly with larger runs. 5. Check whether refreshing the page updates it immediately This is a very useful test: - while the page looks stale, manually refresh the browser - if it updates immediately, that points more to UI polling/staleness - if it is still stale after refresh, that points more to backend/API/database lag Likely recommendations depending on what you find If using SQLite - migrate to PostgreSQL If using PostgreSQL - increase DB pool size / overflow - inspect DB performance - check query latency on graph endpoint - consider whether very large task graphs are making the graph endpoint expensive If the graph endpoint is specifically slow There are settings that can reduce graph load complexity, such as lowering the max graph nodes/artifacts returned. Potential settings to review:
Copy code
PREFECT_SERVER_FLOW_RUN_GRAPH_MAX_NODES
PREFECT_SERVER_FLOW_RUN_GRAPH_MAX_ARTIFACTS
If the page refresh fixes it immediately Then this may be mostly a UI polling/refresh behavior issue rather than a server bottleneck. Practical next steps
Please send these 4 details and I can narrow it down much more concretely: 1. Database type: SQLite or PostgreSQL? 2. Approximate number of task runs in a “slow” flow run 3. Does browser refresh fix it immediately? 4. How long do the browser network requests take for the graph/count endpoints? If helpful, I can also give you a short checklist for: - checking the relevant network requests in the browser - checking the self-hosted server env vars in Kubernetes - tuning Postgres-backed Prefect server for this specific symptom Relevant resources: - Prefect docs - Flows - Work pools
r
@Marvin In Prefect 3.7.0 self hosted kubernetes using sub charts for postgres, redis and other bits, what is a good setting for
Copy code
PREFECT_SERVER_FLOW_RUN_GRAPH_MAX_NODES
PREFECT_SERVER_FLOW_RUN_GRAPH_MAX_ARTIFACTS
and
Copy code
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW
if there are a couple thousand flow runs per day most having, say 20 nodes, but also some with 100 nodes
m
✅ thought for 166.8 seconds
For the workload you described, I would not tune the graph limits aggressively. I’d start by tuning the DB pool modestly and leave the graph limits high unless you know you have unusually huge graphs or artifacts. Recommended starting point
Copy code
PREFECT_SERVER_FLOW_RUN_GRAPH_MAX_NODES=10000
PREFECT_SERVER_FLOW_RUN_GRAPH_MAX_ARTIFACTS=10000

PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=10
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=20
Why these values Graph limits - Your flows are mostly ~20 nodes, with some ~100 nodes - the defaults of 
10000
/ 
10000
are already far above your actual graph sizes - these settings are more like safety ceilings than primary performance knobs - lowering them won’t materially speed up a 20–100 node graph query - lowering them too much can break graph rendering for larger runs later So for your scale, I would leave both graph limits at default. DB pool settings The defaults are: - 
POOL_SIZE=5
- 
MAX_OVERFLOW=10
Those are okay for small usage, but for self-hosted Kubernetes with a couple thousand flow runs/day, a modest increase is reasonable, especially if you have multiple users in the UI and concurrent API activity. A good starting point is:
Copy code
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=10
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=20
That gives the API more headroom for bursts without being too aggressive. Important caveat These are per Prefect server/API instance, so total possible DB connections is approximately:
Copy code
(pool_size + max_overflow) * number_of_api_pods
So if you have 3 API pods:
Copy code
(10 + 20) * 3 = 90
and you should make sure your Postgres 
max_connections
comfortably supports that, plus some margin for Postgres itself and any other clients. If you want a more conservative starting point You could also begin with:
Copy code
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=10
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=10
That’s a bit gentler on Postgres and may already be enough. What I would actually do in your case Given: - couple thousand flow runs/day - most runs 20 nodes - some runs 100 nodes - self-hosted k8s - Postgres already present I’d use:
Copy code
PREFECT_SERVER_FLOW_RUN_GRAPH_MAX_NODES=10000
PREFECT_SERVER_FLOW_RUN_GRAPH_MAX_ARTIFACTS=10000
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=10
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=20
and then observe: - Postgres active connection count - latency for 
/graph-v2
and task count endpoints in browser dev tools - whether page refresh helps immediately - whether Prefect API logs show connection pool waits/timeouts What not to expect Changing 
MAX_NODES
and 
MAX_ARTIFACTS
from 
10000
down to, say, 
500
is unlikely to solve “graph updates are delayed by 10 minutes” when your real runs are only 20–100 nodes. That symptom sounds more like: - DB/API latency - UI refresh behavior - or server-side query contention than a graph-limit ceiling issue. Suggested ranges if you want explicit sizing guidance - Small/medium self-hosted:
Copy code
POOL_SIZE=10
  MAX_OVERFLOW=10
- Moderate with some UI concurrency / bursts:
Copy code
POOL_SIZE=10
  MAX_OVERFLOW=20
- Heavier multi-user install, multiple API pods: maybe go higher, but only after checking Postgres capacity Bottom line For your described workload:
Copy code
PREFECT_SERVER_FLOW_RUN_GRAPH_MAX_NODES=10000
PREFECT_SERVER_FLOW_RUN_GRAPH_MAX_ARTIFACTS=10000
PREFECT_SERVER_DATABASE_SQLALCHEMY_POOL_SIZE=10
PREFECT_SERVER_DATABASE_SQLALCHEMY_MAX_OVERFLOW=20
If you want, I can also help you translate that into the exact Helm 
values.yaml
env block for the self-hosted Prefect chart, and suggest a matching Postgres 
max_connections
target based on how many Prefect API/server pods you run.
r
@Marvin I'm trying your /flow_runs/<id>/graph-v2 thingy and can't figure out what url to type for that in the browser, yo! Can give a bro a worked example, yo?
m
✅ thought for 14.1 seconds
Yep 😄 For a self-hosted Prefect server/UI, the API endpoint is usually under the same base host as the UI, with 
/api
in front. So if your UI is something like:
Copy code
<https://prefect.example.com>
then the graph endpoint for a specific flow run is typically:
Copy code
<https://prefect.example.com/api/flow_runs/<FLOW_RUN_ID>/graph-v2>
Worked example If: - your Prefect UI is at 
<https://prefect.mycompany.net>
- your flow run id is 
12345678-abcd-4321-ef00-1234567890ab
then try:
Copy code
<https://prefect.mycompany.net/api/flow_runs/12345678-abcd-4321-ef00-1234567890ab/graph-v2>
How to get the flow run id Open the flow run page in the UI. The URL will usually include the run id, something like:
Copy code
<https://prefect.mycompany.net/runs/flow-run/12345678-abcd-4321-ef00-1234567890ab>
Then the corresponding API URL is:
Copy code
<https://prefect.mycompany.net/api/flow_runs/12345678-abcd-4321-ef00-1234567890ab/graph-v2>
Quick way to verify the base API path Try this first in the browser:
Copy code
<https://prefect.mycompany.net/api/health>
If that works, your API base is almost certainly:
Copy code
<https://prefect.mycompany.net/api>
Then just append:
Copy code
/flow_runs/<FLOW_RUN_ID>/graph-v2
If your UI and API are on different hosts Sometimes self-hosted setups expose them separately. Then it may be more like:
Copy code
<https://prefect-api.mycompany.net/flow_runs/<FLOW_RUN_ID>/graph-v2>
But in many Helm/chart-based setups it’s:
Copy code
https://<your-prefect-host>/api/...
Even easier method Open browser dev tools on the flow run page: - go to Network - refresh the page - filter for 
graph
- click the request - copy the full request URL That gives you the exact endpoint your UI is calling. If you want, paste: - your Prefect UI base URL pattern, and - one example flow run page URL with the hostname anonymized and I’ll turn it into the exact graph URL format for your setup.
r
@Marvin, yo brah! I tried that and inserted the flow id and still nothing. Sup with dat, yo
m
✅ thought for 16.0 seconds
Heh — fair 😄 If 
/api/flow_runs/<id>/graph-v2
“still nothing,” there are a few likely reasons: 1. Wrong ID Make sure you’re using the flow run ID, not: - flow ID - deployment ID - flow name - task run ID The flow run page URL usually looks like:
Copy code
<https://your-prefect-host/runs/flow-run/aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee>
That last UUID is the one you want. 2. You need the API host/path exactly right Sometimes self-hosted setups are: - 
<https://host/api/>...
- sometimes 
<https://host/prefect/api/>...
- sometimes UI and API are split So the easiest way to get the exact URL is: Open DevTools -> Network -> refresh the flow run page -> filter for 
graph
or 
flow_runs
. Then copy the exact request URL the UI is calling. That is the real answer for your setup. 3. “Nothing” means different things Can you say which one of these happens? - browser shows 
404
- browser shows 
401
or 
403
- browser shows blank page - browser downloads JSON - browser spins forever - browser says 
{"detail":"Not found"}
- browser says internal server error Those mean very different things. 4. You may need query params or auth/cookies Sometimes hitting the endpoint directly in a new tab behaves differently than letting the UI call it with session/auth headers. If you’re logged into the same host in the browser, it often works, but not always depending on ingress/auth setup. Best way to test, bro-style but effective Do this exactly: 1. Open the flow run page in Prefect UI 2. Open browser dev tools 3. Click Network 4. Refresh page 5. In the filter box, type:
Copy code
graph
6. Click the request that looks like 
graph-v2
7. Copy: - Request URL - Status code - maybe Response body Then paste those here. That will tell us whether: - you had the wrong path - the request is failing - the request is slow - or the UI is using a different endpoint than expected Example of what you might discover Maybe your actual request is:
Copy code
<https://prefect.example.com/api/ui/flow_runs/<id>/graph-v2>
or:
Copy code
<https://prefect.example.com/prefect/api/flow_runs/<id>/graph-v2>
or it returns a 500, which would be the real clue. So yeah: DevTools network tab is the move here. Paste me the actual request URL + status code, and I’ll help decode what’s up with dat, yo.
r
@Marvin it sez unauthorised
m
✅ thought for 10.4 seconds
That helps a lot. If it says unauthorized, then the URL is probably basically right, but opening it directly in the browser tab is not sending the auth the same way the UI does. For self-hosted Prefect, that usually means one of these: - your UI session cookie isn’t being used for that direct request - your ingress/auth proxy is protecting 
/api
- the API expects a bearer token/header - UI and API are not actually on the same auth context Fastest way to test anyway Don’t type the URL manually in a fresh tab. Instead use the browser Network tab from the already-loaded Prefect UI page: 1. Open the flow run page in the UI 2. Open DevTools → Network 3. Refresh the page 4. Filter by 
graph
5. Click the 
graph-v2
request 6. Inspect: - Request URL - Status code - Timing - Response That lets you see whether the UI’s own authenticated request is: - succeeding slowly - failing - or not being made at all That’s much more useful than opening the URL directly. If you want to test outside the browser Use the same auth method your self-hosted API expects. For example, if your API accepts a bearer token:
Copy code
curl -H "Authorization: Bearer <TOKEN>" \
  "<https://your-prefect-host/api/flow_runs/<FLOW_RUN_ID>/graph-v2>"
But whether that works depends on how your self-hosted auth is configured. What to look for in DevTools On the 
graph-v2
request, tell me: - the full path used - the status code - the time it took - whether the response is data or an error Same for any task-count request you see. That will tell us whether your slow UI is caused by: - slow API response - auth failures/retries - or frontend refresh behavior So the move now is: don’t open it directly; inspect the authenticated network request the UI itself makes. If you want, I can guide you step-by-step through exactly what to click in Chrome/Firefox DevTools.
2 Views
Open in Slack
PreviousNext
Powered by