What are the upper limits on flow tasks? I want to apply arbitrary algorithms to my flight data(~400k flights). I've been running prefect with PostgreSQL containerized on my pc and it can't cope at higher numbers. Only working with a 10k subset currently. On our academic HPC I'm hoping to see improvement but really hoping there is some guidance for tuning to allow me to take advantage of the scale available.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.