https://prefect.io
Join Slack
I have a fairly hilarious problem that I’m not quite sure how to solve. So I’m running self-hosted P...
# ask-community
r
I have a fairly hilarious problem that I’m not quite sure how to solve. So I’m running self-hosted Prefect, and the UI had become ungodly slow - taking 15-20 seconds or more to resolve sometimes. No worries, still useable, new product, etc, etc Went in to see if it was something I could put in a bugfix for, and I discovered the UI is trying to download about 1GB of data when trying to render the page lol [-> 🧵 ]
Specifically, it’s this UI api call [copied out of devtools as curl command and lightly edited for privacy] 
curl '<https://werehaus.site/api/flow_runs/filter>' \
-H 'authority: <http://werehaus.site|werehaus.site>' \
-H 'accept: application/json, text/plain, */*' \
-H 'accept-language: en-US,en;q=0.9' \
-H 'content-type: application/json' \
-H 'cookie: logged_out_marketing_header_id=blahblah; _ga=blahblah; _ga_blahblah=blahblahblah' \
-H 'origin: <https://werehaus.site>' \
-H 'referer: <https://werehaus.site/flow-runs>' \
-H 'sec-ch-ua: "Not.A/Brand";v="8", "Chromium";v="114", "Google Chrome";v="114"' \
-H 'sec-ch-ua-mobile: ?0' \
-H 'sec-ch-ua-platform: "macOS"' \
-H 'sec-fetch-dest: empty' \
-H 'sec-fetch-mode: cors' \
-H 'sec-fetch-site: same-origin' \
-H 'user-agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/114.0.0.0 Safari/537.36' \
-H 'x-prefect-ui: true' \
--data-raw '{"flow_runs":{"expected_start_time":{"before_":"2023-06-15T06:59:59.999Z","after_":"2023-06-06T07:00:00.000Z"}},"sort":"START_TIME_DESC"}' \
--compressed
terminal crashes when I try to open, so I’m having trouble getting you a sample. It’s an 845MB file containing all of the data files I process daily, captured because they were parameters to a subflow
hang on, let me check that I’m not lying
oh you can pass bytes to 
head
, TIL
so there’s a few normal looking entries for scheduled flows and then this thing
{
"id":"246bba3b-d01b-427a-8e1e-eb59e42d585b",
"created":"2023-06-13T11:34:44.821804+00:00",
"updated":"2023-06-13T11:43:46.647193+00:00",
"name":"curly-impala",
"flow_id":"0a5a23d3-3d0d-4a81-b114-ceb45fad54ca","state_id":"99d54758-53f8-4dfb-af4c-5d51cfb9aeb5",
"deployment_id":null,
"work_queue_id":null,
"work_queue_name":null,
"flow_version":"2d50993280a293fe8a999910e662b216",
"parameters":{
"pages":[
"<?xml version=\"1.0\" encoding=\"UTF-8\" ?>\n<!DOCTYPE 50O MB CHONKYTONK XML FILE...."
and that’s about as far as I felt like paging down
I don’t think this particular thing _need_s to be a subflow (I wrote it fairly early on), so I’m gonna try that
basically the 
pages
parameter is the results of having to crawl a very slow and very terrible API one page at a time, but all at once because they page by document ID and not by anything guessable and also only embed the link to the next page at the end of the previous page. much enterprise. so I’m passing the results of that to a transform subflow
lmk if you want a bug report on github
d
Def. create a bug report if you think its a bug 😄
r
okay! thanks so much lol
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by