Hi I am looking into building pipelines that involve GPU wor

Question

Hi I am looking into building pipelines that involve GPU work on a very irregular basis that autoscaling is a must In my case dask is not really an option as it interferes with the multiprocessing of pytorchs dataloader and otherwise was rather unstable for my GPU workload To make it simpler I would like to start a flow that I know needs GPU on it s own instance and do the processing there The nodes would need to spin up or down depending on current demand Any pointers how to best achieve this Currently I see different possibilities but not sure which one is best Spin up a new Agent on a GPU node before the flow is scheduled and then using tags Somehow use KubernetesRun to request a GPU and let kubernetes handle the up and downscaling Only use prefect to trigger a ECS job

Stephanie Kirmer · Answer

I ve used dask with the dataloader for multiprocessing what trouble did you have there

Kilian · Answer

The issue was with daemonic processes not being able to spawn children

Stephanie Kirmer · Answer

Ah yeah I know that issue for me setting this environment variable fixed it ```DASK DISTRIBUTED WORKER DAEMON=False```

Stephanie Kirmer · Answer

That was a gnarly problem very frustrating

Hugo Shi · Answer

one other thing is that after you do that you should also pass a multiprocessing context into the data loader because the dask multiprocessing context is different than what pytorch typically uses

Hugo Shi · Answer

Something like this ```train loader = torch utils data DataLoader whole dataset sampler=train sampler batch size=batch size num workers=num workers multiprocessing context=mp get context fork ```

Kilian · Answer

Thanks I will try this for sure Any hints for scaling up without Dask slightly smiling face

Stephanie Kirmer · Answer

Sorry we re the dask people slightly smiling face We also helped build a library that made parallel training with dask + pytorch a lot easier if it s any help dask pytorch ddp it s on pypi

Hugo Shi · Answer

we being < Stephanie Kirmer> and me but other folks here may have other thoughts

Hugo Shi · Answer

< Kilian> are you talking about training or inference Or something different alltogether

Kilian · Answer

thanks for the help anyway maybe going this route will also help achieve the endgoal slightly smiling face

Hugo Shi · Answer

if you only need a single GPU instance you don t need multiple GPU instances that need to communicate with each other than prefect + ECS or k8s will probably do just fine without dask

Hugo Shi · Answer

if you re trying to do parallel training with DDP then your PyTorch machines would need some way to coordinate you need to figure out which one is the master and pass that information around

Kilian · Answer

In the end both but one of them would already help Yes only single GPU instance is necessary No parallel training So for that I would use KubernetesRun specify the need for GPU there and let k8s handle up and downscale Thank you for your help