So I've discovered that for any scalability I have to use postgresql but what I've also found is that even if I have a working install in my singularity container that other things can use prefect seems to not accept the configuration.
The prefect docker containers don't seem to have posgresql in them so that assumes a separate postgresql server elsewhere. Orchestration in HPC is not straightforward so I really need to make it work within a single container unfortunately. Bit stuck.๐