https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> are the deploy limits for cpu and ram...
# ask-marvin
o
@Marvin are the deploy limits for cpu and ram the same for docker as for kubernetes pods? job_variables={ "namespace": "namespace", "image_pull_policy": "Always", "finished_job_ttl": 3600, "cpu_request": "128m", "cpu_limit": "16", "memory_request": "128Mi", "memory_limit": "30Gi"
m
✅ thought for 89.7 seconds
Short answer: no — Kubernetes pod resources and Docker container resources are not expressed the same way in Prefect. Your example:
Copy code
python
job_variables = {
    "namespace": "namespace",
    "image_pull_policy": "Always",
    "finished_job_ttl": 3600,
    "cpu_request": "128m",
    "cpu_limit": "16",
    "memory_request": "128Mi",
    "memory_limit": "30Gi"
}
is Kubernetes-specific syntax. What’s different For Kubernetes workers / pods - 
cpu_request
/ 
cpu_limit
- 
memory_request
/ 
memory_limit
These use Kubernetes resource quantity units: - CPU: 
500m
, 
1
, 
2
, etc. - Memory: 
128Mi
, 
1Gi
, etc. So in your example: - 
cpu_request: "128m"
= 0.128 CPU - 
cpu_limit: "16"
= 16 CPUs - 
memory_request: "128Mi"
= 128 MiB - 
memory_limit: "30Gi"
= 30 GiB For Docker workers Docker does not use those same fields. Instead, Prefect’s Docker worker supports: - 
mem_limit
- 
memswap_limit
and CPU controls need to go through: - 
container_create_kwargs
So the equivalent shape is more like:
Copy code
python
job_variables = {
    "mem_limit": "30g",
    "container_create_kwargs": {
        "cpu_quota": 1600000,
        "cpu_period": 100000
    }
}
or other Docker CPU options like: - 
cpu_shares
- 
cpuset_cpus
- 
cpu_quota
- 
cpu_period
Important unit difference Memory units differ too: - Kubernetes: 
Mi
, 
Gi
- Docker: typically 
m
, 
g
, 
k
, bytes So: - Kubernetes 
128Mi
- Docker roughly 
128m
Those are similar in intent, but not the same format. Bottom line - Kubernetes: requests + limits, K8s units - Docker: hard limits only for memory, Docker runtime-style CPU settings via 
container_create_kwargs
- So you should not expect 
cpu_request
, 
cpu_limit
, 
memory_request
, 
memory_limit
to work the same for Docker deployments If you want, I can translate that exact Kubernetes 
job_variables
block into the closest valid Docker worker 
job_variables
equivalent.
o
@Marvin why 1600000 cpu period 10000
m
✅ thought for 6.0 seconds
Because Docker CPU limits are usually expressed as:
Copy code
text
usable CPUs = cpu_quota / cpu_period
So if you want to allow 16 CPUs, you set quota to 16 times the period. Example with a common 
cpu_period
of 
100000
microseconds:
Copy code
python
cpu_quota = 16 * 100000 = 1600000
cpu_period = 100000
That means the container can use 1,600,000 microseconds of CPU time every 100,000 microseconds per CPU scheduling period, which effectively caps it at 16 CPUs. So the example should be:
Copy code
python
"container_create_kwargs": {
    "cpu_quota": 1600000,
    "cpu_period": 100000
}
not 
10000
. Equivalent examples - 1 CPU → 
cpu_quota=100000
, 
cpu_period=100000
- 2 CPUs → 
cpu_quota=200000
, 
cpu_period=100000
- 0.5 CPU → 
cpu_quota=50000
, 
cpu_period=100000
- 16 CPUs → 
cpu_quota=1600000
, 
cpu_period=100000
So the general formula is:
Copy code
text
cpu_quota = desired_cpus * cpu_period
If you want, I can give you the full Docker equivalent of your K8s resource block: - 
cpu_request: 128m
- 
cpu_limit: 16
- 
memory_request: 128Mi
- 
memory_limit: 30Gi
with notes on what cannot be represented in Docker.
o
@Marvin and which of the job_variables docker settings is most similar to the mem and cpu limit in kubernetes?
m
✅ thought for 7.5 seconds
The closest Docker equivalents are: - Kubernetes 
memory_limit
 → Docker 
mem_limit
- Kubernetes 
cpu_limit
 → Docker 
container_create_kwargs.cpu_quota
+ 
container_create_kwargs.cpu_period
Example:
Copy code
python
job_variables = {
    "mem_limit": "30g",
    "container_create_kwargs": {
        "cpu_quota": 1600000,
        "cpu_period": 100000
    }
}
What does not map cleanly - Kubernetes 
memory_request
 → no real Docker equivalent - Kubernetes 
cpu_request
 → no real Docker equivalent That’s because Kubernetes has a request vs limit model: - 
request
= scheduling guarantee / reservation - 
limit
= hard cap Docker mostly gives you runtime constraints, not scheduler reservations. Rough mapping table - 
memory_limit
→ 
mem_limit
- 
cpu_limit
→ 
cpu_quota
+ 
cpu_period
- 
memory_request
→ no equivalent - 
cpu_request
→ no equivalent If you want a softer CPU control in Docker, 
cpu_shares
is also related, but it is not the same as a Kubernetes CPU limit. It’s more of a relative weighting than a hard cap. If you'd like, I can turn your exact Kubernetes block into the best-practice Docker worker job_variables version.
o
@Marvin so then 1600000 / 100000 to 16?!?
m
✅ thought for 6.8 seconds
Yes — exactly.
Copy code
text
1600000 / 100000 = 16
So with:
Copy code
python
"container_create_kwargs": {
    "cpu_quota": 1600000,
    "cpu_period": 100000
}
Docker interprets that as 16 CPUs worth of runtime. Intuition - 
cpu_period
= length of one scheduling window, in microseconds - 
cpu_quota
= how much CPU time the container can consume during that window - 
cpu_quota / cpu_period
= number of CPUs the container may use Examples:
Copy code
text
100000 / 100000 = 1 CPU
50000 / 100000 = 0.5 CPU
200000 / 100000 = 2 CPU
1600000 / 100000 = 16 CPU
So it’s just a ratio. One caution: this is the closest Docker equivalent to Kubernetes 
cpu_limit
, but it’s still Docker runtime semantics, not Kubernetes scheduling semantics. If you want, I can give you the exact Docker 
job_variables
for your original K8s values: - 
cpu_limit: "16"
- 
memory_limit: "30Gi"
o
@Marvin why do the
Copy code
"container_create_kwargs": {
    "cpu_quota": 1600000,
    "cpu_period": 100000
}
do not appear in the run job varibales, althoufh visiable in the base job template of the docker deployment?
m
✅ thought for 122.7 seconds
Yes — that’s expected. Why this happens 
run job variables
usually show only the overrides stored on the deployment or flow run, not the fully resolved worker configuration. So if you put:
Copy code
python
"container_create_kwargs": {
    "cpu_quota": 1600000,
    "cpu_period": 100000
}
in the work pool’s base job template, those values are inherited as defaults. They may not appear in the run’s displayed 
job_variables
unless they were explicitly set as overrides on that deployment or on that specific flow run. Think of it like this There are three layers: 1. Work pool base job template - default config for all runs in that pool 2. Deployment 
job_variables
 - deployment-specific overrides 3. Flow run 
job_variables
 - run-specific overrides The worker merges them at runtime into one final config. So: - visible in base job template → yes - visible in run job variables → only if explicitly overridden there - still applied at runtime → yes Why 
container_create_kwargs
feels confusing For Docker, 
cpu_quota
and 
cpu_period
live nested under 
container_create_kwargs
, and those are merged into the actual Docker container create call internally by the worker. The resolved config is used by the worker, but that fully merged object is not typically what the UI shows as 
job_variables
. In short If it’s only set in the base job template, it can be: - applied - but not shown in run job variables If you want it to show up in run job variables, set it explicitly in the deployment’s 
job_variables
, e.g.:
Copy code
python
job_variables = {
    "mem_limit": "30g",
    "container_create_kwargs": {
        "cpu_quota": 1600000,
        "cpu_period": 100000
    }
}
Most likely explanation in your case You’re seeing: - 
container_create_kwargs
in the Docker work pool base template - but not in the run details because Prefect is showing you the overrides, not the resolved inherited defaults. If you want, I can also show you exactly where in the Docker worker config those values get merged into the actual Docker container settings.
o
@Marvin whats the most used pattern to export the mem and cpu profiles of docker containers to grafana?
m
✅ thought for 65.4 seconds
The most common pattern is: 
cAdvisor + Prometheus + Grafana
 That’s the standard setup for per-container CPU and memory monitoring in Docker. Why this is the usual choice - 
cAdvisor
collects CPU / RAM / filesystem / network metrics for each container - 
Prometheus
scrapes and stores those metrics over time - 
Grafana
visualizes them and lets you alert on them Best fit for your use case If you want CPU and memory profiles for Prefect flow-run containers launched by a Docker worker, this is usually the best pattern. Typical architecture
Copy code
text
Docker host
 ├─ Prefect worker
 ├─ Flow run containers
 ├─ cAdvisor
 └─ Prometheus

Grafana
 └─ reads from Prometheus
Why not just Docker metrics? You can use Docker daemon metrics, but they’re usually more limited for detailed per-container dashboards. Why not node_exporter alone? 
node_exporter
is great for host metrics, not detailed container-level CPU/RAM usage. Why not OpenTelemetry first? OpenTelemetry is better for application-level telemetry and tracing. If your main question is container CPU/RAM, it’s usually overkill compared with cAdvisor. Prefect-specific note Prefect Docker flow-run containers include labels you can use to identify them in monitoring. That makes it easier to build Grafana panels per flow run or deployment. A useful pattern is to filter metrics by container labels/names that identify Prefect-created containers. Common metrics people graph From cAdvisor / Prometheus, typically: - CPU usage rate - memory working set - memory usage / limit - container restarts - network I/O - filesystem usage Examples:
Copy code
text
rate(container_cpu_usage_seconds_total[5m])
container_memory_working_set_bytes
container_memory_usage_bytes
Practical recommendation If you want the most widely used, low-friction stack: 1. Run 
cAdvisor
2. Scrape it with 
Prometheus
3. Build dashboards in 
Grafana
Starter docs - Prometheus + cAdvisor guide - cAdvisor - Grafana dashboards If you want, I can also give you a minimal 
docker-compose.yml
for: - 
cAdvisor
- 
Prometheus
- 
Grafana
and show how to filter Prefect flow-run containers specifically.
o
@Marvin must i integrate the memory or cepy u profiles per flow ? or where is the client that connects to the grafana dashboard?
m
✅ thought for 7.8 seconds
No — you do not instrument CPU or memory per flow for this pattern. There is usually no client inside the flow sending container CPU/RAM to Grafana. How it works instead For container infrastructure metrics, the flow container is just observed from the outside: 1. 
cAdvisor
runs on the Docker host 2. 
cAdvisor
reads container stats from Docker / cgroups 3. 
Prometheus
scrapes 
cAdvisor
4. 
Grafana
queries 
Prometheus
So the “client” is effectively: - 
cAdvisor
exposing metrics - 
Prometheus
scraping them - 
Grafana
reading from Prometheus Your Prefect flow code does not need to push CPU/memory metrics. So where does the Grafana connection happen? Grafana usually connects to Prometheus as a data source. Not: - flow → Grafana Usually: - container stats → cAdvisor → Prometheus → Grafana What you need to run Typically on the Docker host or in the same environment: - 
cAdvisor
- 
Prometheus
- 
Grafana
When would you instrument inside the flow? Only if you want application-level metrics, such as: - rows processed - task duration - API latency - custom counters - business KPIs Then you might add: - Prometheus client library - StatsD - OpenTelemetry But for: - CPU usage - RAM usage - container limits - container lifecycle you usually do not add code to the flow. Rule of thumb - Container/host metrics → external monitoring stack - Business/app metrics → instrument inside the flow In your case Since you asked about Docker container CPU/RAM profiles, the normal setup is: - no per-flow integration required - monitor the Prefect-created containers from the host If you want, I can draw the exact setup for a Prefect Docker worker host and provide a minimal 
docker-compose.yml
for 
cAdvisor + Prometheus + Grafana
.
o
@Marvin provide a minimal example 
cAdvisor + Prometheus + Grafana
m
✅ thought for 8.1 seconds
Here’s a minimal local setup with 
cAdvisor + Prometheus + Grafana
. Files 
docker-compose.yml
Copy code
yaml
services:
  cadvisor:
    image: <http://gcr.io/cadvisor/cadvisor:latest|gcr.io/cadvisor/cadvisor:latest>
    container_name: cadvisor
    ports:
      - "8080:8080"
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker:/var/lib/docker:ro
    restart: unless-stopped

  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
    restart: unless-stopped
    depends_on:
      - cadvisor

  grafana:
    image: grafana/grafana:latest
    container_name: grafana
    ports:
      - "3000:3000"
    restart: unless-stopped
    depends_on:
      - prometheus
prometheus.yml
Copy code
yaml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: cadvisor
    static_configs:
      - targets: ["cadvisor:8080"]
Start it
Copy code
bash
docker compose up -d
Open the UIs - cAdvisor: http://localhost:8080 - Prometheus: http://localhost:9090 - Grafana: http://localhost:3000 Grafana login - username: 
admin
- password: 
admin
Add Prometheus as a Grafana data source In Grafana: - Connections / Data Sources - Add data source - Choose 
Prometheus
- URL: 
<http://prometheus:9090>
- Save & test If 
<http://prometheus:9090>
doesn’t work in your setup, use the hostname reachable from the Grafana container/network. Example Prometheus queries in Grafana CPU usage per container:
Copy code
text
rate(container_cpu_usage_seconds_total[5m])
Memory working set:
Copy code
text
container_memory_working_set_bytes
Memory usage in MiB:
Copy code
text
container_memory_working_set_bytes / 1024 / 1024
CPU by container name:
Copy code
text
sum by (name) (rate(container_cpu_usage_seconds_total[5m]))
Memory by container name:
Copy code
text
sum by (name) (container_memory_working_set_bytes)
Useful note for Prefect Docker workers Your flow runs are just Docker containers, so they should show up automatically in cAdvisor metrics. Then in Grafana you can filter by container name/labels to isolate Prefect runs. Common next improvement Add 
node_exporter
too if you also want host-level metrics, but for a minimal example, the above is enough. Docs: - Prometheus cAdvisor guide - cAdvisor - Grafana docs If you want, I can give you a second version with: - persistent Grafana storage - preconfigured Prometheus datasource - example Grafana dashboard JSON - filtering specifically for Prefect containers
10 Views
Open in Slack
PreviousNext
Powered by