https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> I've got API/SERVER metrics enabled (...
# ask-marvin
m
@Marvin I've got API/SERVER metrics enabled (by setting 
PREFECT_API_ENABLE_METRICS
to 
true
). But on my local prefect server, when I open the 
/metrics
endpoint, I only see metrics on system resources (like filedescriptors, garbage collection and cpu_seconds), but I'm not seeing any metrics on number of flow runs. How do I get those metrics?
m
✅ thought for 186.6 seconds
Short answer: you don’t get flow-run-count metrics from 
/metrics
by default. With 
PREFECT_API_ENABLE_METRICS=true
, Prefect’s 
/metrics
endpoint exposes mostly: - Python/runtime metrics like GC, CPU, file descriptors - process metrics - some Prefect event/log client metrics But not built-in Prometheus metrics like “number of flow runs”. If you want flow run counts, the native way is to query the API directly, for example:
Copy code
bash
curl -X POST <http://localhost:4200/api/flow_runs/count>
You can also filter that count via the API if you only want certain flows, deployments, states, etc. So the distinction is: - 
/metrics
= Prometheus/process-style telemetry - 
/api/flow_runs/count
= Prefect operational count data If your goal is to put flow-run counts into Prometheus/Grafana, you’d typically do one of these: 1. Poll Prefect’s count endpoints and export them yourself 2. Build a small custom exporter that calls: - 
/api/flow_runs/count
- optionally 
/api/task_runs/count
3. Scrape that exporter from Prometheus Relevant endpoint: - Prefect docs If helpful, I can give you a small example exporter script that turns Prefect flow run counts into Prometheus metrics.
m
@Marvin FYI the problem was caused by the increase in required disk storage for the static frontend files. Our docker image has a read-only filesystem, so for the frontend files we have to explicitly mount a volume. We had the volume configured to 20MB, but the required size with the new UI update went from ~17MB to ~53MB. Just letting you know this was causing the restart issues. The lack of error messages is a bit unfortunate.
m
✅ thought for 16.9 seconds
Thanks — that’s a really useful finding. So the root cause was: - your Docker image uses a read-only filesystem - Prefect needed writable space for the frontend static assets - the mounted volume was only 20MB - after the UI update, the frontend assets grew from about 17MB to about 53MB - that caused the server restart issues And agreed: the lack of a clear error message here is unfortunate, especially since it can look unrelated to storage at first. For anyone else hitting something similar, the practical fix is to ensure the writable mount for Prefect’s frontend/static files is sized generously enough for UI assets after upgrades. I couldn’t find an existing GitHub issue or discussion covering this exact case. If you want, I can help you turn this into a concise GitHub issue report so the Prefect team/community has a searchable record of it.
m
@Marvin yes please draft a github issue I can post
m
✅ thought for 18.6 seconds
Absolutely — here’s a draft you can post as a GitHub issue in PrefectHQ/prefect issues. Title Prefect server can restart/fail silently when frontend static asset volume is too small on read-only filesystems Body
Copy code
text
### Bug summary

When running Prefect server in Docker with a read-only root filesystem, Prefect may repeatedly restart or fail to come up if the writable volume used for frontend static assets is too small.

In our case, this happened after a UI update increased the required disk space for static frontend files. The configured writable volume was previously large enough, but after the update it was no longer sufficient.

The main difficulty was that the failure mode did not produce a clear error message indicating that disk space for frontend assets was the issue.

### Version info

Please fill in:
- Prefect version: 
- Deployment method: Docker
- OS / platform: 
- Python version: 

### Expected behavior

If Prefect cannot write or unpack required frontend static files because the writable filesystem/volume is too small, it should fail with a clear and actionable error message indicating:
- which path needs writable storage
- how much space was required / available, if possible
- that this may happen in read-only filesystem container setups

### Actual behavior

Prefect server restarted / failed to start, but without a sufficiently clear error message pointing to insufficient disk space for frontend static assets.

### Environment / setup

We run Prefect server in a Docker container with:
- read-only root filesystem
- an explicit writable mounted volume for frontend/static files

That writable volume was configured to 20MB.

Before the UI update, frontend files required about 17MB, so the setup worked.

After the UI update, the required size increased to about 53MB, and the server began restarting.

### Steps to reproduce

A rough reproduction path:

1. Run Prefect server in Docker with a read-only root filesystem
2. Mount a small writable volume for frontend/static files
3. Limit that volume to something below the required static asset size (for example ~20MB)
4. Start or upgrade Prefect to a version with larger frontend assets
5. Observe startup failure or restart loop

### What we found

The root cause was insufficient writable disk space for frontend static files.

Increasing the mounted volume size resolved the restart issue.

### Suggested improvement

It would help if Prefect:
- logged a clear startup error when frontend assets cannot be written due to insufficient space
- documented writable filesystem requirements for containerized read-only deployments
- possibly documented expected frontend asset storage requirements or recommended buffer space across upgrades

### Additional context

In our case, the frontend asset size increased from roughly 17MB to roughly 53MB after a UI update, which exceeded the configured writable volume size.
If you want, I can also make this more concise and “maintainer-friendly” before you post it.
m
@Marvin well, yes, make it as concise and developer friendly as possible. Please also make sure it mentions the "Server Stopped!" info-level log line which was the only visible feedback which other people who run into the same issue might use to search for issues
m
✅ thought for 9.6 seconds
Here’s a tighter, developer-friendly version: Title Prefect server restart loop on read-only filesystem when frontend asset volume is too small; only visible log is 
Server Stopped!
Body
Copy code
text
### Bug summary

When running Prefect server in Docker with a read-only root filesystem, the server can fail to start / enter a restart loop if the writable volume used for frontend static assets is too small.

In this failure mode, the only visible log we saw was:

`Server Stopped!`

That made the issue difficult to diagnose and also makes it hard for other users to search for.

### Environment

- Deployment method: Docker
- Filesystem setup: read-only root filesystem
- Writable mount: separate volume for frontend/static files
- Writable volume size: 20MB

### Root cause

A recent UI update increased the required space for frontend static assets from roughly 17MB to roughly 53MB.

Our mounted writable volume was still sized for the older asset footprint, so Prefect could no longer write/unpack the frontend files and the server restarted.

Increasing the volume size resolved the issue.

### Expected behavior

If Prefect cannot write frontend static assets because the writable filesystem is too small, startup should fail with a clear, actionable error message indicating:
- the path that needs writable storage
- that disk space is insufficient
- ideally the required vs available space

### Actual behavior

The server stopped/restarted, and the only visible feedback was the info-level log line:

`Server Stopped!`

### Suggested improvement

Please surface a more specific startup error when frontend asset initialization fails due to insufficient writable storage, especially for read-only container deployments.
You can post it here: - PrefectHQ/prefect issues If you want, I can also draft a 1-paragraph GitHub Discussion version in case you’d rather post it as a discussion than an issue.
Open in Slack
PreviousNext
Powered by