https://prefect.io logo
Join Slack
Powered by
Hi all, I’m experiencing an issue with passing a ...
# ask-community
c
Hi all, I’m experiencing an issue with passing a large parameter in a flow that leads to the UI becoming unresponsive and crashing. There is discussion here specific to dataframes, https://github.com/PrefectHQ/prefect/issues/6962, and a fix in PR that will use orjson, but the issue with the UI crashing is not specific to dataframes, and any large parameter (e.g. array with 1MM elements) would cause the same issue. How Prefect is designed, the parameters passed to a flow are retrieved as part of the flow runs api. If a parameter produces a large string when serialized, this will led to a massive response coming back and either causing a timeout, or crashing the server dependent on the size. A workaround here is to avoid passing large parameters, write to disk/s3 and then pass the URL as the parameter. This has the downside increasing runtime and creating an IO bottleneck. Curious on the communities / Prefect teams thoughts about this paradigm? Wanted to start a discussion here as it is technically not a bug and working as intended, but has side-effects in standard use-cases. Additional information added in reply.
Manual Repro:
Copy code
curl '<http://0.0.0.0:4200/api/flow_runs/filter>

{"id":"86793b9b-b6a9-44cf-b98a-ef29692eac8a","type":"COMPLETED","name":"Completed","timestamp":"2022-11-03T06:10:09.397238+00:00","message":"All states completed.","data":null,"state_details":{"flow_run_id":"5ae1d25a-60eb-4209-97a8-941a449f9be9","task_run_id":null,"child_flow_run_id":null,"scheduled_time":null,"cache_key":null,"cache_expiration":null,"untrackable_result":false}}},{"id":"5a229192-34c2-41f8-a866-e46c6b02d960","created":"2022-11-03T04:26:30.684953+00:00","updated":"2022-11-03T04:26:43.320342+00:00","name":"nickel-caterpillar","flow_id":"d6830ecb-4841-4987-be8c-b11cee7dab1e","state_id":"6f3e4d33-e804-4dac-bd6b-059f21326c13","deployment_id":null,"work_queue_name":null,"flow_version":"ad34935ca7bb0ce1597b101badd96d93","parameters":{"job_id":"6dd18125-ba62-4a42-bf18-fad91b94d89e","device_paths":[{"device_id":"redacted","file_path":"redacted"},{"device_id":"redacted","file_path":"redacted"}],"input_bucket":"redacted","output_bucket":"redacted","report_meta_data":{"device_ids":["redacted"],"company_name":"redacted","collection_id":null},"historical_trend_data_json_str":"{\"tenant_id\":{\"0\":\"16\",\"1\":\"16\",\"2\":\"16\",\"3\":\"16\",\"4\":\"16\",\"5\":\"16\",\"6\":\"16\",\"7\":\"16\",\"8\":\"16\",\"9\":\"16\",\"10\":\"16\",\"11\":\"16\",\"12\":\"16\",\"13\":\"16\",\"14\":\"16\",\"15\":\"16\",\"16\":\"16\",\"17\":\"16\",\"18\":\"16\",\"19\":\"16\",\"20\":\"16\",\"21\":\"16\",\"22\":\"16\",\"23\":\"16\",\"24\":\"16\",\"25\":\"16\",\"26\":\"16\",\"27\":\"16\",\"28\":\"16\",\"29\":\"16\",\"30\":\"16\",\"31\":\"16\",\"32\":\"16\",\"33\":\"16\",\"34\":\"16\",\"35\":\"16\",\"36\":\"16\",\"37\":\"16\",\"38\":\"16\",\"39\":\"16\",\"40\":\"16\",\"41\":\"16\",\"42\":\"16\",\"43\":\"16\",\"44\":\"16\",\"45\":\"16\",\"46\":\"16\",\"47\":\"16\",\"48\":\"16\",\"49\":\"16\",\"50\":\"16\",\"51\":\"16\",\"52\":\"16\",\"53\":\"16\",\"54\":\"16\",\"55\":\"16\",\"56\":\"16\",\"57\":\"16\",\"58\":\"16\",\"59\":\"16\",\"60\":\"16\",\"61\":\"16\",\"62\":\"16\",\"63\":\"16\",\"64\":\"16\",\"65\":\"16\",\"66\":\"16\",\"67\":\"16\",\"68\":\"16\",\"69\":\"16\",\"70\":\"16\",\"71\":\"16\",\"72\":\"16\",\"73\":\"16\",\"74\":\"16\",\"75\":\"16\",\"76\":\"16\",\"77\":\"16\",\"78\":\"16\",\"79\":\"16\",\"80\":\"16\",\"81\":\"16\",\"82\":\"16\",\"83\":\"16\",\"84\":\"16\",\"85\":\"16\",\"86\":\"16\",\"87\":\"16\",\"88\":\"16\",\"89\":\"16\",\"90\":\"16\",\"91\":\"16\",\"92\":\"16\",\"93\":\"16\",\"94\":\"16\",\"95\":\"16\",\"96\":\"16\",\"97\":\"16\",\"98\":\"16\",\"99\":\"16\",\"100\":\"16\",\"101\":\"16\",\"102\":\"16\",\"103\":\"16\",\"104\":\"16\",\"105\":\"16\",\"106\":\"16\",\"107\":\"16\",\"108\":\"16\",\"109\":\"16\",\"110\":\"16\",\"111\":\"16\",\"112\":\"16\",\"113\":\"16\",\"114\":\"16\",\"115\":\"16\",\"116\":\"16\",\"117\":\"16\",\"118\":\"16\",\"119\":\"16\",\"120\":\"16\",\"121\":\"16\",

... continues for 16MBs worth of a response
Loading on the flow runs instantly kills my container as it’s trying to load 500Mb worth of responses
Some load balancer stats
a
can you upgrade to 2.6.6? I think we addressed the issue in this latest release today
A workaround here is to avoid passing large parameters
another option is to pass it as data dependency and use some database/data store to retrieve the value?
I think if you pass large objects as parameter values, then the use case lends itself more to data dependency within a flow (dataflow use case, not parametrization use case) there is some ambiguity though e.g. with subflows so let me tag @Zanie and see if he has some opinions on what's the right approach here from a backend perspective
c
@Anna Geller I’ve updated to 2.6.6, but still experiencing the issue. The previous runs were created on 2.6.5 though, so not sure if that has an impact on the 2.6.6 potential fix.
I think for our use-case, we’d probably opt to rewrite to have one large flow with no subflows to keep it all in-memory with the tradeoffs of losing retries / other subflow/task behavior.
👍 1
z
I think we should probably just drop large parameters from the API payload
👍 1
2 Views
Open in Slack
PreviousNext
Powered by