Anyone else running prefect via singularity on aca...
# show-and-tellj
Joni Pelham
08/08/2023, 8:18 PMAnyone else running prefect via singularity on academic HPC? I'm using the Dask-MPI backend. https://github.com/jonititan/FDMSingularity
🙌 3
✅ 1
j
Jonathan Pelham
08/26/2023, 12:58 PMSo I've discovered that for any scalability I have to use postgresql but what I've also found is that even if I have a working install in my singularity container that other things can use prefect seems to not accept the configuration.
The prefect docker containers don't seem to have posgresql in them so that assumes a separate postgresql server elsewhere. Orchestration in HPC is not straightforward so I really need to make it work within a single container unfortunately. Bit stuck.😅
Jonathan Pelham
08/26/2023, 3:05 PMNow no matter how i change my container prefect throws an error.
RuntimeError: Cannot create flow run. Failed to reach API at http://127.0.0.1:4200/api/.
🤨
Jonathan Pelham
08/27/2023, 12:34 PMwelp i've managed to fix the issue. Prefect stores config in a .prefect folder in my home directory so no matter what I did with the image because singularity loads your home folder by default it always took its config from there.
I'll need to work out how to do a bind or something to put it somewhere more tightly linked to my singularity builds.
Now I just need to work out how to get the postgresql backend working. Once I've done that I should be able to scale my workflows up to actually take advantage of the HPC and not just fail where the default backend(sqlite) gets overwhelmed from concurrent writes.
Jonathan Pelham
08/29/2023, 2:33 PMNow have a working example for running prefect with postgres running as a service both using singularity. https://github.com/jonititan/FDMSingularity
Currently working out how to improve running
11 Views