<@ULVA73B9P> I have the following flow which works...
# marvin-ai
l
@Marvin I have the following flow which works against the same server when run locally and fails with
UnboundLocalError: cannot access local variable 'batch' where it is not associated with a value
when i run it through the deployment which uses docker and kubernertes worker:
Copy code
import json
from datetime import datetime

from dateutil import parser
from freight_request_processor import get_distance_mapping, process_single
from freight_request_repository import save_requests_pg
from more_itertools import chunked
from prefect import flow, get_run_logger, task, unmapped
from prefect.blocks.system import Secret
from prefect.cache_policies import NO_CACHE
from prefect.task_runners import ThreadPoolTaskRunner
from timocom import Timocom
from utils import (
    build_pg_utils_from_block,
    get_config,
    parse_location_criteria,
    publish_refresh_message,
)

@flow(task_runner=ThreadPoolTaskRunner())
def timocom_search_flow(
    search_parameters: dict,
    pg_secret_block: str,
    timocom_secret_block: str,
    rabbitmq_secret_block: str,
    ors_server_variable: str,
    price_calculation_server_variable: str,
    predictive_scoring_server_variable: str,
    here_maps_api_key_variable: str,
):
    logger = get_run_logger()
    <http://logger.info|logger.info>(f"Starting Timocom search with parameters: {search_parameters} at {datetime.now()}")

    try:
        sanitized_search_parameters = sanitize_and_validate_search_parameters(search_parameters)
        pg_utils = build_pg_utils_from_block(pg_secret_block)

        # Perform Timocom search

        # Timocom credentials are stored as "tenant_id": {"user": "username", "pass": "password"}
        creds = Secret.load(timocom_secret_block).get()
        tenant_ref = sanitized_search_parameters["tenant_ref"]
        <http://logger.info|logger.info>(f"Using Timocom credentials for tenant: {tenant_ref}")
        tenant_creds = creds.get(tenant_ref)
        if not tenant_creds:
            raise ValueError(f"No Timocom credentials found for tenant {tenant_ref}")

        user = tenant_creds["user"]
        password = tenant_creds["pass"]

        # Initialize Timocom instance
        timocom_instance = Timocom(logger, pg_utils, user, password)
        # Create search tasks
        search_tasks = timocom_instance.create_search_tasks(sanitized_search_parameters)
        search_futures = timocom_instance.search.map(search_tasks)

        <http://logger.info|logger.info>("Timocom search process completed.")

        config = get_config(
            ors_server_variable,
            price_calculation_server_variable,
            predictive_scoring_server_variable,
            here_maps_api_key_variable,
        )

        for future in search_futures:
            for batch in chunked(future.result(), 50):
                <http://logger.info|logger.info>(f"Processing batch of {len(batch)} results...")

                distance_mapping = get_distance_mapping(batch, config)

                proccessed_results = process_single.map(unmapped(config), batch, distance_mapping)
                <http://logger.info|logger.info>("Post-processing of freight requests completed.")

                save_requests_pg(proccessed_results, logger, pg_secret_block)

                <http://logger.info|logger.info>("Requests data persistence completed.")
                <http://logger.info|logger.info>(f"Publish message: {datetime.now()}")
                publish_refresh_message(
                    rabbitmq_secret_block,
                    search_parameters["tab_id"],
                    search_parameters["job_id"],
                    "REFRESH",
                )

    except Exception as e:
        logger.error(f"Error during Timocom search initiation: {e}")
        publish_refresh_message(
            rabbitmq_secret_block, search_parameters["tab_id"], search_parameters["job_id"], "FAILED"
        )
        raise
m
thought for 360.6 seconds
Thanks for sharing the flow and error. That
UnboundLocalError: cannot access local variable 'batch' where it is not associated with a value
is a Python scoping error (not Prefect-specific). It usually happens when: - a loop like
for batch in ...:
never runs (e.g., empty iterable), and some code path still tries to use
batch
later, or - an exception occurs before the first iteration and an error handler/log statement references
batch
, or - a closure/comprehension captures
batch
in an unexpected way. A few concrete things I’d tighten up in your flow that will both avoid the error and make mapping behave correctly in Prefect 3.x: 1) Materialize your batches before looping and handle the empty case - If
future.result()
returns an empty list, your
for batch in ...
loop won’t run and any accidental reference to
batch
will blow up. Materializing the chunks first makes this explicit and safer. 2) Wrap non-mapped arguments with
unmapped(...)
- In
process_single.map(unmapped(config), batch, distance_mapping)
, Prefect may treat
distance_mapping
(a dict) as an iterable and try to map it too. If you intend to use the same
distance_mapping
for all items in the current batch, wrap it with
unmapped(distance_mapping)
. 3) Resolve mapped results before passing to a plain Python function - If
save_requests_pg
is not a Prefect task, pass it concrete results with
.result()
instead of a
PrefectFutureList
. Here’s a revised flow excerpt applying those points:
Copy code
for future in search_futures:
    results = future.result()  # resolve each search task
    batches = list(chunked(results, 50))

    if not batches:
        <http://logger.info|logger.info>("No results returned for this search task; skipping.")
        continue

    for i, batch in enumerate(batches, start=1):
        <http://logger.info|logger.info>(f"Processing batch {i}/{len(batches)} with {len(batch)} results...")

        distance_mapping = get_distance_mapping(batch, config)

        # Ensure only `batch` is mapped; config and mapping are static per item
        processed_futures = process_single.map(
            unmapped(config),
            batch,
            unmapped(distance_mapping),
        )

        # If save_requests_pg is a normal function, resolve mapped results first
        processed_results = processed_futures.result()
        save_requests_pg(processed_results, logger, pg_secret_block)

        <http://logger.info|logger.info>("Requests data persistence completed.")
        <http://logger.info|logger.info>(f"Publish message: {datetime.now()}")
        publish_refresh_message(
            rabbitmq_secret_block,
            search_parameters["tab_id"],
            search_parameters["job_id"],
            "REFRESH",
        )
Why this likely fixes your issue - If
future.result()
returns no items in your K8s/Docker environment (but did locally), your original
for batch in ...
loop would never bind
batch
. Any reference to
batch
elsewhere (even unintentionally via a log/closure/comprehension/etc.) triggers the UnboundLocalError. The pattern above avoids that by checking
if not batches: continue
. -
unmapped(distance_mapping)
ensures Prefect doesn’t try to map over the dict’s keys, which can lead to odd behavior and mismatched mapping lengths. A couple of quick checks that will help us confirm root cause: - Could you share the full traceback (including the line number where
UnboundLocalError
is raised)? - What Python version are you running locally vs. in your Docker image? - Which Prefect version are you on (assuming 3.x)? And is
save_requests_pg
a plain function or a Prefect task? If you prefer, we can also encapsulate per-batch work into a dedicated task (e.g.,
@task def process_batch(...)
) and map batches directly. That gives you cleaner concurrency and easier debugging in the UI.