<@ULVA73B9P> We have a work pool configured as typ...
# ask-marvin
m
@Marvin We have a work pool configured as type "docker", which runs workflows using locally-built docker images. When running a workflow that wraps a "dbt run" command in a ShellOperation, we've noticed that the container never really goes above 1 CPU in use, even though the dbt project is configured to use 32 threads. The server is not constrained in any way (CPU and RAM usage are both sitting at around 2.5%). Are there any default resource limits on flows run using a Docker work pool? Or is wrapping the shell script that runs dbt in a ShellOperation somehow limiting resource usage?
m
It appears that Prefect's documentation doesn't explicitly mention default resource limits for Docker work pools at a glance. However, managed execution with Prefect Cloud seems to require running the official Prefect Docker image,
prefecthq/prefect:2-latest
, but additional package dependencies can be installed at runtime (Managed Execution - Prefect Docs). It's also worth mentioning that if you need to use your own image, managed execution with the official Docker image might not be suitable, and another type of work pool may be recommended. The tutorial section about workers also doesn't provide specifics on default resource constraints within Docker work pools, just the higher-level setup instructions (Work Pools and Workers - Prefect Docs). The GitHub search didn't return any relevant open issues about Docker work pool resource limits. Considering the information discovered, your issue may not be due to default resource limits set by Prefect. It could be related to the actual configuration of the Docker container or the Docker daemon on the host system. You might want to inspect the Docker daemon's configs to ensure it's not limiting CPU resources for containers. Additionally, ensure that your dbt project configuration is aligned with the Docker settings. Furthermore, reviewing Docker's own resource allocation settings, such as CPU quotas and periods for the container, might provide clarity. If you're still unable to resolve the issue, it might be beneficial to directly reach out to Prefect's support or community channels for guidance.