Dear Community, We have a running prefect server version 2.13.0 hosted on an Azure VM, with several deployments which are running as expected on CPU VMs. We have been using prefect agent to run these deployments in Docker images (with the Docker Infrastructure block) with no problems. These same images also have GPU based code which I am able to execute properly through:

docker run --gpus=all --entrypoint python <IMAGE URL> path/to/python/file.py

. Yet when I try to run the same (GPU bound) python file using a prefect agent, the container is run without the

--gpus=all

switch and lacks GPU support, which renders the code unusable. We have tried running the prefect agent with 'NVIDIA_VISIBLE_DEVICES=all' as an enviroment variable both from the CLI and as a systemd service, to no avail. Note that this env. var. does not replace the need for '--gpus=all' switch. Your help is truly appreciated🙏

Is something new in Your case ? I have very similar problem to resolve.