Hi community,
what would be a suggested/recommended setup to be used with azure as host and Prefect as keystone ; about 10 python scripts are executed daily 8 are lightweight and 2 of them do a lot of heavy lifting , they are changed with lots of frequency so maintenance overall cant be too complicated but for cost optimization some sort of elasticity would be most appreciated as the requirements of the heavy scripts are 10x the small scripts hardware reqs.
thanks a bunch in advance
✅ 1
b
Bianca Hoch
01/06/2023, 7:23 PM
Hi Nicolas, I'm not terribly familiar with Azure, but I can point you in the direction for tutorials and guides for Creating a Prefect 2.0 PoC on Azure.
Bianca Hoch
01/06/2023, 7:26 PM
In regard to the flows changing with high frequency, here are some resources for CI/CD that may be of use too.
Bianca Hoch
01/06/2023, 7:27 PM
I'm sure that others who are more familiar with Azure will be able to share their thoughts here as well. 😄
n
Nicolas Gastaldi
01/06/2023, 9:07 PM
Hi Bianca , thank you so much for the insight, i will look into it ; i made a basic poc using a single vm and docker , it worked very well for my requirements so i wanted to give it a try on Prefect when it gets real “firepower” 😄 ; hence the question, if VM strategy or ScaleSets or the Container CI as suggested in the tutorials would be best fight for heavy workload
thanks again for the directions , i will look into it
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.