Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
o
Oleg Sheyner
07/29/2022, 5:29 PMHi everyone, evaluating Prefect and hoping to get your advice on getting to a working cloud POC as fast as possible. Our situation:
• Mostly Windows/Azure shop. Preference to run on Windows, all else equal, but can work with Linux VMs if necessary.
• We have nothing in our ecosystem yet running on VMs (serverless only) and no Docker/Kubernetes setup.
How would you suggest we set up a POC that deploys and runs flows on Azure and connects to Prefect Cloud:
• POC should include demo for publishing and running flows to Azure from local dev machines
• Primary goal is to set up as fast as possible:
• Set up on a bare VM? Install and run under Docker? Try Azure AKS? (no experience at present with Kubernetes/AKS, but can be good case to acquire such experience in-house)
Thanks much in advance!
👀 4
m
Mason Menges
07/29/2022, 6:59 PMHey @Oleg Sheyner Not super familiar with Azure But I'd start by checking out these docs they should help get you in the right direction.
https://orion-docs.prefect.io/concepts/storage/
https://docs.prefect.io/concepts/deployments/
https://docs.prefect.io/concepts/infrastructure/
c
Chris Reuter
07/29/2022, 7:15 PM@Christopher Boyd is our resident Azure pro and may have some advice
c
Christopher Boyd
07/29/2022, 7:49 PMHey @Oleg Sheyner - I think setting up through AKS isn’t too much trouble, and I have some public recipes in prefecthq/prefect-recipes to deploy an AKS cluster and the prefect agent / environment.
With that said, it’s not up to date with GA and I need to update it. I have some additional content that I’m working on for deploying the server / agent to Kubernetes. For the most part, it’s pretty straightforward to get a basic setup through running the prefect agent, generating a manifest, and applying it via kubectl apply -f
https://github.com/PrefectHQ/prefect-recipes/tree/main/azure/prefect-2/prefect-agent-on-aks
https://discourse.prefect.io/t/prefect-1-0-prefect-2-0-kubernetes-run-config-to-kubernetesjob-infrastructure-block/1229
Notably, the prefect recipe I linked, you can use as is for the terraform aspect just to deploy your infrastructure.
The prefect steps themselves (steps 6 and on in the readme) I need to go back over and confirm they are still accurate, but our documentation should be sufficient to apply the manifest from there
At a high level it would involve:
Deploying your cluster
Configuring your storage; if using 2.0, unfortunately we don’t have Azure blocks available just yet but are in progress.
Starting your agent - this can be run either locally, or in-cluster. Local is simple to setup; in-cluster I have a document that is a work in progress , but there is an example manifest I shared yesterday to get it running.
Creating your deployment
Running your deployment
o
Oleg Sheyner
07/30/2022, 3:17 AM@Christopher Boyd Sounds great, will review. In the meantime, to cover all possible avenues - what about deploying an image on a single Windows Server 2019 VM with Docker? Have people tried this, any writeups?