https://prefect.io logo
Join Slack
Powered by
can you export logs from a run in prefect cloud? O...
# ask-community
a
can you export logs from a run in prefect cloud? Or, more specifically I want to copy logs including timestamp, level, and logger - if you just drag-select it only copies the messages
n
hi @Alastair - yes you can; logs for the entire run are available for download as a csv from the overflow menu in the top right of the page
a
ah, found it thanks @nicholas. I wasn't looking that far "outside" the logs view
n
yeah i think we should put that button a little closer to the logs themselves!
a
it doesn't include events, does it?
that'd be super useful to be able to get too
(I'm trying to debug an issue wehre a run has gone into Crashed but still seems to be doing stuff)
‼️ 1
n
ahh it does not include events, i'm not sure we have export capabilities around those at the moment
i'll create an issue internally to track that as a request though, i can see why that'd be helpful
a
ok, claude made me a cli tool to get them, so I'm good for now
💙 1
thanks!
blob attention gif 1
ok, not sure if this should go in another thread, but maybe you could help...
👀 1
Copy code
2026-03-09 19:47:45 | ERROR    | prefect_cloud.push_work_pool | Could not load job configuration: Client error '429 Too Many Requests' for url '<http://orion-internal:4200/api/accounts/x/workspaces/y/block_documents/z?include_secrets=true>'
For more information check: <https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/429>
2026-03-09 19:47:45 | EVENT    | prefect.flow-run.Crashed
we've been seeing this a bit - it seems your internal code flows also use our rate limits to get stuff, which... sure is a decision, but I guess we can live with that
so I have a top level flow which is running, and we call run_deployment, it looks like the internal infra looks up our aws keys to set up the ecs push run, but hits the rate limit. not great... but then... it switches the entire (parent) flow run to crashed? not sure if I'm reading this correctly
n
hmm i'm not sure why that error would be getting thrown from an internal service, that doesn't seem correct. i'll raise that internally to see what could be going on there.
a
I was a bit surprised to see your internal infra leaked into our logs!
hard same 1
n
on your other question though, can you share the URL for a flow run where this was happening?
a
-> DM
smart 1
n
feel free to share that via DM if you're more comfrtable beat me to it
a
what we're trying to do: parent flow does some setup kicks off x subdflows we sit around polling for them to finish once they;re done we do some more orchestration
what happens: kick off subflows eventually go over a rate limit (we're trying to resolve this) everything turns to Crashed???
n
and you'd expect that if one of the subflows crashes, the parent flow run would handle that crash as a regular failure, right?
a
is there some magic propagation of subflow status to parent status that I've missed?
n
no not at all - just regular python error raising in this case
a
we're using run_deployment for the subflows (we want them on a separate infra)
n
thank you @Alastair! i'm not sure what's up here but have raised this with the team to see if they know what's going on here - will report back as soon as i know more
a
just to clarify - specifically about the unusual state changing? 🙃
not sure how many different things we're tracking here
just looping in @Micheál Keane , as it's the end of my day
n
2 things in particular, for your acute problem: 1. why are sub flow run crashes propagating to the parent 2. what's going on with that internal service rate limit message - it might be a correct rate limit instance but it's certainly not a correct error response
a
great, thanks. I did see one subflow crash, I think. is that what you're thinking is happening here?
btw, the new rate limiting info page in the account section is much much better than the old one
but we still have no idea what requests of ours are turning into either API requests or Events
just as this error started to happen, we saw a huge increase in API requests - from my understanding this was just at the moment that all of our subflows finished
so I'm not sure if it was an unexpected request storm in our 
wait_for_flows_to_finish
function, or something internal, or...
z
Hey @Alastair! A few things on rate limits
but we still have no idea what requests of ours are turning into either API requests or Events
Really appreciate this feedback, thank you! I'll update the copy to clarify • Events include the number of logs, task run events, and custom events created by your account • API Requests includes requests to all orchestration API endpoints. (Effectively all API endpoints except the control plane endpoints like 
GET /api/accounts/{:id}
) There's more detail in the docs https://docs.prefect.io/v3/concepts/rate-limits#api-rate-limits On the API Requests chart, there's a dropdown to breakdown usage by route, which will show you everything that's included. Looking at your usage, it seems light other than heavy usage of concurrency limits
a
thanks @Zach Angell, some feedback on the docs: • You say "Number of logs" - I guess you mean "number of log messages emitted" - given that I've seen some logs emitted by the prefect infrastructure in our messages, are they also counted? • I don't know what a "task run event" is, the phrase doesn't occur anywhere else on that page or on https://docs.prefect.io/v3/concepts/events. Is this the change in event state of a single @task? Is this for events that are automatically generated by prefect (Pending, Running, Completed) for every single task or just for ones that we emit manually? • Has the grouping of rate limits changed with the new updates? In our previous email thread with support we were told that there was one limit for API requests (flows, flows runs, deployments, tasks, etc), and a separate one for "orchestration" (which included concurrency tag lookups). Are they now all bundled together? • "and other resources" is doing a lot of heavy lifting in that bullet point, especially if it includes "this results in at least 2 requests every time you use a tag on a task if it has a concurrency value set", In our opinion this is still very poorly documented
View your account’s rate limit usage in the Prefect Cloud UI under Account Settings > Rate Limits.
This link doesn't exist
z
You say "Number of logs" - I guess you mean "number of log messages emitted" - given that I've seen some logs emitted by the prefect infrastructure in our messages, are they also counted?
Yes
don't know what a "task run event" is, the phrase doesn't occur anywhere else on that page or on https://docs.prefect.io/v3/concepts/events. Is this the change in event state of a single @task? Is this for events that are automatically generated by prefect (Pending, Running, Completed) for every single task or just for ones that we emit manually?
Yep one event for every state transition of a task. Automatically generated.
Has the grouping of rate limits changed with the new updates? In our previous email thread with support we were told that there was one limit for API requests (flows, flows runs, deployments, tasks, etc), and a separate one for "orchestration" (which included concurrency tag lookups). Are they now all bundled together?
Yes previously limits applied independently to endpoint groups, resulting in confusion. For simplicity they are now all bundled together
"and other resources" is doing a lot of heavy lifting in that bullet point, especially if it includes "this results in at least 2 requests every time you use a tag on a task if it has a concurrency value set", In our opinion this is still very poorly documented
I'll add a callout for this, thanks!
This link doesn't exist
Whoops! Tracked down the bug and will get that fixed
a
thanks for the clarifications here @Zach Angell, IMO those would be really great to put on the rate limiting page in order to make it clearer/more explicit as to whats going on
it seems that these conditions have changed quite significantly since the last support email I sent, especially the fact that task runs and state transitions now count towards the limits, I don't think they did previously
we've previoulsy had quite an issue with hitting the limits due to getting/releasing concurrency tags - now that they're combined with other API requests, we saw yesterday that basically us doing too many concurrency checks caused other parts of the system to unexpectedly fail too. I mentioned this in the "and other resources" comment, I think it'd be nice to clearly call out how many requests a rate limit check takes
I've had a bit of a learning experience today - we're still quite disappointed with the limits available for concurrency tags. the (new??) limits for state transitions fall into this category a bit too - It really seems here like there's a "prefect tax" on running python functions - especially as we're running on our own ECS infra - basically we're unable to make the most of our resources because we have to limit them to stay under our rate limits
our current architecture is that we have a scrape/analysis pipeline, so we might be getting 500-5000 urls and for each of them we do an HTTP GET, and then another loop where we make an LLM request with the content. In our current setup, the http get and llm process functions are decorated as @tasks. when we first started with prefect we were happy to see the concurrency limits because we thought that we could use prefect to orchestrate our parallelism and also ensure we stay under the rate limits of the external services
what we saw though was that we could never hit the limits of our external services because we hit the prefect rate limit first
after a bunch of investigation today, chat basically told me that we were doing it wrong, and that most orchestration platforms kind of expect you to manually batch stuff - so instead of fanning out on 2k http requests, you fan out on 20 batches of 100 items each with a bit of magic to work out how to do work inside each batch. Tbh this is something new to me! I kind of just assumed that the primitives would allow me to fan out work naively, and there wasn't really anything in the docs that seemed to indicate otherwise
anyway - maybe there is one specific question here: do you see people using prefect for similar tasks as I've described, and do they typically use this batching pattern?
other than that, mostly just observations about our process - I wonder if things could be made clearer in some way to help others avoid the same mistakes
z
Hey @Alastair thanks again for all your feedback here it's super helpful! Sorry for the radio silence, not ignoring just working out the best path forward.
anyway - maybe there is one specific question here:
do you see people using prefect for similar tasks as I've described, and do they typically use this batching pattern?
Yep people do use prefect for similar tasks. At a certain scale rate limiting ends up guiding people to use this batching pattern. As you've pointed out, forcing users to batch instead of using primitives isn't ideal at the scale you're running (500-5000 urls). We're figuring out the best way to improve the experience here
a
thanks for the response @Zach Angell. Glad to hear the feedback is being taken onboard
11 Views
Open in Slack
PreviousNext
Powered by