That payload means Prefect did persist a result — and the persisted result value is an empty dict. Your example:

{"metadata":{...},"result":"{}"}

means: -

metadata

= how Prefect stored the result -

result

= the serialized value -

"{}"

= the serialized representation of an empty dictionary So the main takeaway is: Prefect thinks the final returned value was

{}

. Common reasons: 1. The object you returned was actually empty at return time

@flow
def my_flow():
    data = {}
    return data

2. You returned a wrapper object instead of the actual data For example, returning a future/state-like object instead of its resolved value can lead to confusing persisted output. Example:

@flow
def my_flow():
    future = my_task.submit()
    return future.result()   # good

instead of:

@flow
def my_flow():
    future = my_task.submit()
    return future            # not the actual dict

3. You expected an upstream task’s dict, but that task itself returned

{}

This is very common if some transformation/filter step removed all keys before the return. 4. You are looking at the serialized payload rather than the deserialized result Prefect stores results in a serialized envelope. In storage/API/raw views, you may see:

{"metadata": ..., "result": "{}"}

but the meaningful part is still just the deserialized result:

{}

. A few important notes from your payload: - The serializer is

json

- That means Prefect encoded the result as JSON - If your dict had content, you’d expect something like:

"result":"{\"foo\":1,\"bar\":\"x\"}"

- Since it is exactly

"{}"

, the serialized value itself was empty What to check next Add a log right before the return:

from prefect import flow, get_run_logger

@flow
def my_flow():
    logger = get_run_logger()
    result = some_function()
    <http://logger.info|logger.info>("Result before return: %r", result)
    return result

If this logs

{}

, then Prefect is behaving correctly. If you want, paste the flow/task code where you produce and return that dict, and I can help pinpoint why it became empty.

Thanks — that’s a very useful clue. If your log right before return shows a large non-empty dict, but the persisted payload is still:

{"metadata": ..., "result":"{}"}

then this is not just “you returned an empty dict”. Most likely, one of these is happening: 1. The value you logged is not the exact value Prefect persisted This can happen if: - you log

results

- then later mutate

results

- and finally return the mutated object Example:

<http://logger.info|logger.info>(f"Returned results: {results}")
results.clear()
return results

Because dicts are mutable, logging earlier does not guarantee the returned object stayed the same. 2. Serialization is failing or coercing something unexpectedly Your logged object includes a very large nested GeoJSON-like structure. With the

json

serializer, Prefect needs the returned object to be fully JSON-serializable. From your example, this part is notable: - nested large dicts/lists - link-like objects - possibly custom classes or mappings mixed into plain dicts Even if the top-level object prints like a dict, some nested values may not serialize cleanly. 3. Something is wrapping/replacing the returned object before persistence This can happen if you: - manually create/return

State

objects - have custom hooks/callbacks touching the returned data - are returning a result wrapper instead of plain Python data 4. You may be inspecting the wrong stored result If this flow returns multiple task results or subflow results, the storage object you’re looking at may correspond to a different run/result than the logged object. What I’d do next First, log immediately at return and log the exact type:

<http://logger.info|logger.info>("Type of results: %s", type(results))
<http://logger.info|logger.info>("Results repr right before return: %r", results)
return results

Second, test plain JSON serialization yourself before returning:

import json

json.dumps(results)
<http://logger.info|logger.info>("JSON serialization succeeded")
return results

If this raises, you’ve found the cause. Third, make a deep copy at return time to rule out mutation:

import copy

final_results = copy.deepcopy(results)
<http://logger.info|logger.info>("Final results right before return: %r", final_results)
return final_results

Fourth, try reducing the payload Return just a small known-good subset:

return {
    "super-flow": results["super-flow"]
}

If that persists correctly, then the problematic part is likely inside

heat_regions

. Fifth, confirm whether this is a flow result or a task result Because Prefect stores them separately, it matters whether you are inspecting: - the flow run result - a task run result - a subflow result The logged structure looks like a composed object, so this distinction matters a lot. What the current evidence suggests Because your log clearly shows a non-empty dict, I’d prioritize these hypotheses in order: 1. the dict is mutated after logging but before return 2. a nested value inside the dict is not being serialized as expected 3. you are inspecting a different persisted result than the one produced by that return statement If you want, paste the exact function body from: - where

results

is built - where it is logged - where it is returned - and how result persistence is configured and I can help narrow it down very quickly.

This usually means the thing polling for scheduled work is getting interrupted during HPA events. Short version: if you're running your flow with

flow.serve()

in a Kubernetes Deployment and then letting HPA scale that Deployment, that is not a good fit. In Prefect 3.x,

serve

is meant for a long-lived process on relatively static infrastructure, not a horizontally scaled replicated service. A few important points: Why runs become

Late

- Prefect marks a run

Late

when it stays in

Scheduled

past its expected start time long enough. - If your serving pod is terminating, unready, restarting, or temporarily not polling the Prefect API, scheduled runs can sit unclaimed. - Even if one pod is still alive in Kubernetes, that does not guarantee it is still healthy and polling Prefect. Why HPA +

flow.serve()

is problematic - Multiple replicas running

flow.serve()

are not the recommended HA/scaling pattern. - During scale up/down, pods can briefly become unready or terminate, creating a polling gap. -

flow.serve()

also has

pause_on_shutdown=True

by default, which can cause extra confusion during shutdown events. Verified signatures: -

Flow.serve(..., pause_on_shutdown: bool = True, webserver: bool = False, ...)

-

Flow.deploy(..., work_pool_name=..., image=..., ...)

Recommended pattern instead Use: -

flow.deploy(...)

- a work pool - one or more Prefect workers in Kubernetes - put HPA on the worker deployment, not on a

flow.serve()

app That is the supported elastic-scaling model in Prefect 3.x. Example direction:

python
from prefect import flow

@flow
def my_flow():
    print("hello")

if __name__ == "__main__":
    my_flow.deploy(
        name="my-deployment",
        work_pool_name="my-k8s-pool",
        image="your-registry/your-image:latest",
        cron="*/5 * * * *",
    )

Then run workers separately in K8s, scaling those:

bash
prefect worker start --pool my-k8s-pool --with-healthcheck

Useful docs: - Workers - Work Pools - Run flows on Kubernetes - Run flows in local processes (`serve`) - Worker healthchecks If you must stay on

serve

for now A few things to check immediately: - Are you running more than 1 replica of the serving app? - Is

pause_on_shutdown

still

True

? - Are readiness/liveness probes too aggressive? - Is

terminationGracePeriodSeconds

long enough? - During scale-down, does the remaining pod continue polling, or does it go unready first? If you're using

serve

, I would at minimum try: - only 1 replica -

pause_on_shutdown=False

- avoid HPA on that Deployment - use a separate worker-based architecture for scalable execution Example:

python
my_flow.serve(
    name="my-deployment",
    cron="*/5 * * * *",
    pause_on_shutdown=False,
)

But again, that is more of a mitigation than a real fix. My best guess about your specific symptom The likely sequence is: 1. HPA triggers a scaling event 2. the serving pod(s) become unready / terminate / stop polling 3. scheduled runs are created but not picked up in time 4. Prefect marks them

Late

5. Kubernetes still shows one pod alive, but that pod is not actually serving Prefect work correctly at that moment If you want, I can help you debug this more concretely if you share: - whether you are using

flow.serve()

or workers - your Deployment/HPA YAML - probe settings - replica count - whether

pause_on_shutdown

is set - Prefect version and worker type if applicable