<@ULVA73B9P> Hey everyone, About two hours ago, ...
# ask-marvin
d
@Marvin Hey everyone, About two hours ago, one of my deployments started consistently failing with the following error: ‘’’ prefect.exceptions.PrefectHTTPStatusError: Client error '422 Unprocessable Entity' for URL 'https://api.prefect.cloud/api/accounts/*****/workspaces/******/flow_runs/' Response: { 'exception_message': 'Invalid request received. Please check your input for NULL characters or values that may be outside normal ranges.' } More info: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/422 ‘’’ When I run the flow locally against the Prefect Cloud API, I get the same error. But when I run it using a local Prefect server, everything works as before. Has anything changed on Prefect Cloud recently? And does anyone know how to fix or troubleshoot this? Thanks in advance!
m
thought for 551.2 seconds
Short answer: a 422 from the Cloud API means the request body you’re sending has something Cloud now refuses to accept. The error text is the giveaway: a NULL character in a string somewhere, or a value outside normal ranges (e.g., NaN/Infinity, an out‑of‑range datetime). Your local server is more permissive; Cloud is stricter. I don’t see a public note about a breaking change in Cloud in the last releases, but the behavior aligns with recent API hardening/validation. In other words: nothing obviously “down” on Cloud, but your payload now trips validation. - Prefect releases: https://github.com/PrefectHQ/prefect/releases - Cloud troubleshooting: https://docs-3.prefect.io/v3/how-to-guides/cloud/troubleshoot-cloud What to check first - Parameters: Any floats that can be NaN/Infinity? Any bytes-like values that sneak in? Extremely large or deeply nested structures? - Strings: run_name, tags, parameter strings, labels, environment-derived values (branch names, commit messages, dataset fields) containing control chars, especially the null byte \x00. - Datetimes: scheduled or parameter datetimes far outside typical ranges (year < 1 or > 9999), or unusual tz offsets. Quick isolation steps 1) Try a zero-parameter run via CLI and add parameters back until it fails: - Verify syntax: - Run: prefect deployments run --help - Minimal run: - prefect deployments run <FLOW_NAME>/<DEPLOYMENT_NAME> --watch - Then add parameters with -p key=value incrementally to find the offender: - prefect deployments run <FLOW_NAME>/<DEPLOYMENT_NAME> -p some_number=1 -p flag=true --watch 2) If you build run_name dynamically, sanitize it and tags: - Remove control characters: replace any char in [\x00-\x1f] with a safe placeholder. 3) Pre-flight validate your parameters locally before creating the run: - This catches NaN/Infinity and null bytes. Example helper to find bad values - Drop this near where you assemble parameters/run_name/tags. import math from datetime import datetime def find_bad_values(obj, path="root"): issues = [] def walk(x, p): if isinstance(x, str): if "\x00" in x: issues.append(f"{p}: contains NULL byte") # Optional: flag other control chars if any(ord(c) < 32 for c in x): issues.append(f"{p}: contains control characters") elif isinstance(x, float): if math.isnan(x) or math.isinf(x): issues.append(f"{p}: non-finite float (NaN/Inf)") elif isinstance(x, datetime): if not (1 <= x.year <= 9999): issues.append(f"{p}: datetime out of range: {x!r}") elif isinstance(x, dict): for k, v in x.items(): walk(v, f"{p}.{k}") elif isinstance(x, (list, tuple, set)): for i, v in enumerate(x): walk(v, f"{p}[{i}]") walk(obj, path) return issues # Usage examples: # issues = find_bad_values(parameters) # issues += find_bad_values({"run_name": run_name}) # issues += find_bad_values({"tags": list(tags)}) # print(issues) 4) Force strict JSON validation to smoke out NaN/Infinity before Prefect sends it: import json json.dumps(parameters, allow_nan=False) # Will raise ValueError if NaN/Infinity present 5) Turn on HTTP debugging to see the outgoing payload (careful: may include secrets): - In your flow entrypoint: import logging logging.basicConfig(level=logging.DEBUG) logging.getLogger("httpx").setLevel(logging.DEBUG) If none of the above reveals it - Share your Prefect version (I’ll assume 3.x unless you say otherwise), the exact fields you populate (run_name, tags, parameters), and whether anything in your upstream data changed “two hours ago” (new dataset values often carry hidden control chars). - If you can reproduce with a minimal param dict that fails, post it here (scrub secrets). That makes it much easier to confirm which field Cloud is rejecting.
If this is blocking and you’ve isolated a minimal payload that triggers the 422, open a ticket or post on Prefect Slack with the payload shape and your workspace/flow run link so the Cloud team can confirm which validator is firing. In the meantime, sanitizing control characters and replacing NaN/Infinity with None usually gets you unblocked.