Hi I am using DockerRun for my Flow's environment and I am running into some issues. The Docker image pulls and runs fine, but it uses pytorch and I am starting to run into some shared memory issues for my dataloader. Normally using Docker, people have solved this by passing arguments to increase the shared memory size to the docker run command. Is there any way to change the shared memory of the DockerRun on Prefect?

Hey @Nelson Griffiths, was looking at some stuff. So the Docker Agent uses

create_container

from the Docker SDK. I’m not seeing an option here immediately. How would you do it with the

docker run

command?

docker run -it --shm-size=256m

This would change it from 64mb to 256mb

Yeah. Here is the right link. Not seeing anything. Also saw this with attaching gpus where it’s not available in the SDK

I have already tried that. It didn't do anything

Oh it’s under the

create_host_config

. Then that needs to go into

create_container

. I’ll look into this more.

It is exposed on the build side only to provide shared memory to intermediate layers. It returns to the default once the container is running

Gotcha. Thanks for clarifying. Ok will see how we can get this in

That would be awesome. Thank you!

We have the host_config exposed so I think it’s doable. Just need to figure out the syntax.

Oh great. Thank you!

Actually it doesn’t look at that

shm

is configurable. I think we may need to open an issue

Okay. Like I said, happy to help if needed

I can write it then tag you in a bit

Awesome. Thank you so much!

https://github.com/PrefectHQ/prefect/issues/4773 comment here?

Great. I will leave a comment on it. Thank you!