1 year ago
    Hey prefect community, we have a somewhat philosophical question and a question of our fundamental understanding of prefect and orchestration (using dask): How do you approach the infrastructure setup, i.e. selection of node size, number of workers etc.? In a nutshell: • EKS vs Fargate vs others • fewer big nodes with high number of workers vs. many smaller nodes with lower number of workers • how many workers per node in general • how do you optimize between tasks, workers and nodes? What better questions to ask? How do you think and iterate over the infrastructure, prefect, dask setup, and so on? 🙂
    PS: trigger for these questions are heartbeat problems that we encounter which might be due to the ressources we select for our EKS nodes. In the ideal world, one would not have to think about ressources at all, because prefect/coiled/dask/xyz adapt on that automagically, right?