Jonathan Pelham

08/30/2023, 10:32 AM
What are the upper limits on flow tasks? I want to apply arbitrary algorithms to my flight data(~400k flights). I've been running prefect with PostgreSQL containerized on my pc and it can't cope at higher numbers. Only working with a 10k subset currently. On our academic HPC I'm hoping to see improvement but really hoping there is some guidance for tuning to allow me to take advantage of the scale available.