https://prefect.io logo
Join Slack
Powered by
This message was deleted.
# ask-marvin
s
This message was deleted.
m
Your message was too long by 941 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:
on a self-hosted prefect3 server I'm getting a lot of cpu usage and the following errors
File "/usr/local/lib/python3.12/contextlib.py", line 754, in aexit raise exc_details[1] File "/usr/local/lib/python3.12/contextlib.py", line 737, in aexit cb_suppress = await cb(*exc_details) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/contextlib.py", line 217, in aexit await anext(self.gen) File "/usr/local/lib/python3.12/site-packages/prefect/server/events/ordering.py", line 201, in preceding_event_confirmed await handler(waiter, depth + 1) File "/usr/local/lib/python3.12/site-packages/prefect/server/events/triggers.py", line 483, in reactive_evaluation async with AsyncExitStack() as stack: ^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/contextlib.py", line 754, in aexit raise exc_details[1] File "/usr/local/lib/python3.12/contextlib.py", line 737, in aexit cb_suppress = await cb(*exc_details) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/contextlib.py", line 217
It seems you're encountering a "MaxDepthExceeded" error on your self-hosted Prefect 3.x server, which is accompanied by high CPU usage. While direct documentation on this specific error is not readily available, I found some general information about managing events and their limits on a Prefect server. In Prefect 3.x, there are several settings related to events that could be relevant to your issue: 1. maximum_labels_per_resource: The default maximum is 500. 2. maximum_related_resources: Also defaults to 500. 3. maximum_size_bytes: The maximum size of an event when serialized to JSON is 1,500,000 bytes. These settings can be configured in your Prefect server. You might want to look into adjusting them to see if it alleviates the CPU usage problem. The relevant environment variables for these settings are: - 
PREFECT_SERVER_EVENTS_MAXIMUM_LABELS_PER_RESOURCE
- 
PREFECT_EVENTS_MAXIMUM_LABELS_PER_RESOURCE
- 
PREFECT_SERVER_EVENTS_MAXIMUM_RELATED_RESOURCES
- 
PREFECT_EVENTS_MAXIMUM_RELATED_RESOURCES
- 
PREFECT_SERVER_EVENTS_MAXIMUM_SIZE_BYTES
- 
PREFECT_EVENTS_MAXIMUM_SIZE_BYTES
Here's the link to the Prefect documentation where you can explore more about configuring your self-hosted server: Self-Hosting Prefect. Sometimes, the mysteries of the universe come down to tweaking some environment variables. If the issue persists, you might consider reaching out to the community for more insights or sharing logs for a deeper dive.
Your message was too long by 495 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:
on a self-hosted prefect3 server I'm getting a lot of cpu usage and the following errors
File "/usr/local/lib/python3.12/site-packages/prefect/server/events/ordering.py", line 176, in preceding_event_confirmed raise MaxDepthExceeded(event) (edited) prefect.server.events.ordering.MaxDepthExceeded: occurred=DateTime(2024, 12, 6, 9, 40, 27, 780035, tzinfo=Timezone('UTC')) event='prefect.flow-run.Cancelling' resource=Resource(root={'prefect.state-name': 'Cancelling', 'prefect.state-type': 'CANCELLING', 'prefect.resource.id': 'prefect.flow-run.da524876-ba4d-486c-af60-270c49f1cebd', 'prefect.resource.name': 'apricot-pronghorn', 'prefect.state-message': 'The parent flow run was cancelled.', 'prefect.state-timestamp': '2024-12-06T094027.780035+00:00'}) related=[RelatedResource(root={'prefect.resource.id': 'prefect.flow.642333b1-5fc2-4896-a5fb-d02c65e50fdd', 'prefect.resource.name': 'rfd', 'prefect.resource.role': 'flow'}), RelatedResource(root={'prefect.resource.id': 'prefect
2 Views
Open in Slack
PreviousNext
Powered by