Hey prefect community,
we have a somewhat philosophical question and a question of our fundamental understanding of prefect and orchestration (using dask):
How do you approach the infrastructure setup, i.e. selection of node size, number of workers etc.?
In a nutshell:
• EKS vs Fargate vs others
• fewer big nodes with high number of workers vs. many smaller nodes with lower number of workers
• how many workers per node in general
• how do you optimize between tasks, workers and nodes?
What better questions to ask?
How do you think and iterate over the infrastructure, prefect, dask setup, and so on? 🙂