https://prefect.io logo
Title
r

Ryan Peden

10/21/2022, 4:11 AM
As promised in Anna's release Prefect 2.6.4 release announcement, I just published a post introducing the new Azure Container Instances infrastructure block and showing how to set it up to run your flows: https://medium.com/the-prefect-blog/serverless-prefect-flows-with-azure-container-instances-f2442ebc9343
:blob-attention-gif: 6
🙌 9
:azure: 5
:thank-you: 4
👏 4
:prefect: 5
💯 9
❤️ 5
🔥 6
o

Oscar Björhn

10/21/2022, 10:03 AM
This looks really neat! Time for us to start thinking about whether we actually need Kubernetes...
👍 3
🙌 1
💯 2
j

Jeremiah

10/21/2022, 2:27 PM
🤫
t

Thomas Frederik Hoeck

10/24/2022, 8:55 AM
Does this create a ACI resource per deployment? Or can they use the same ACI resource? @Ryan Peden
r

Ryan Peden

10/24/2022, 5:04 PM
Hi Thomas, it creates a container group with a single container per flow run. This makes it easy to change infrastructure parameters like CPU, memory, and GPU without needing to redeploy anything. If you want something that stays running on ACI and runs flows as they come in, you might prefer running an agent on ACI and setting up your flows to run as processes in the agent's container: https://discourse.prefect.io/t/how-to-run-a-prefect-2-agent-in-an-azure-container-instance/1427 If there are other ACI usage patterns you think make sense for Prefect flows, we'd love to hear more about them.
t

Thomas Frederik Hoeck

10/25/2022, 7:21 AM
Thank you from the reply @Ryan Peden. From my understanding a container group can't spread across more than one host node, so won't this potentially put a upper limit on how many flows that can run in parallel with this infra? https://learn.microsoft.com/en-us/azure/container-instances/container-instances-region-availability#linux-container-groups Another pattern could be to have a 1:1 mapping between a Deployment in prefect and a ACI resource. Then you know that there is at least that amount of resources for the flow.
r

Ryan Peden

10/25/2022, 12:36 PM
Hi Thomas, it's a container group + container per individual flow run, not per flow, so having multiple flows (or even multiple runs of one flow) running in parallel should only be limited by the quotas listed on this page - and with a paid subscription, you can request increases to those.
t

Thomas Frederik Hoeck

10/26/2022, 7:49 AM
Oh I thought it was only possible to have one container group per ACI resources. Thank you for clarifying.
i

Ivan Hubysh

01/16/2023, 4:00 PM
Hi @Ryan Peden. Thank you for great article. Unfortunately I am struggling second day to set it up: the healthcheck flow run stays in "late" state forever e.g. no container was created. Also I can not see any logs anywhere. Do you have some tip of how to debug it?
r

Ryan Peden

01/16/2023, 4:03 PM
Hi, Ivan! When I've seen that happen it's often because something goes wrong in the container before the flow run can start. A couple of questions: • Are you using the default image, or a custom image? • Does anything show up in your agent logs?
i

Ivan Hubysh

01/16/2023, 4:04 PM
I tried both with my own on ACR and with default one. Currently staying on the default. Logs are empty
r

Ryan Peden

01/16/2023, 4:08 PM
Are you able to access the machine where the agent runs and see the agent's terminal output? It should be writing messages to the terminal when it picks up the flow run and tries to start an ACI container group to run the flow.
i

Ivan Hubysh

01/16/2023, 4:11 PM
Probably I missed something (it is my second week doing with Prefect). I assumed that deployments would be run on Container Instances triggered directly from Prefect Cloud. So one still needs an Agent/VM, that listens to a queue and fires az cli commands under the hood to spin up containers? upd. Now it makes sense to me. Thanks)
r

Ryan Peden

01/16/2023, 4:25 PM
That's right, although we use the Azure Python SDK instead of running CLI commands. It's part of Prefect's hybrid model. Prefect Cloud orchestrates your flow runs, but all the execution happens on VMs or hardware you control. I wrote a short tutorial on how to run an agent on ACI if you want to run everything there: https://discourse.prefect.io/t/how-to-run-a-prefect-2-agent-in-an-azure-container-instance/1427
❤️ 1
i

ivan

01/16/2023, 7:54 PM
Thank you, Ryan.
i

Ivan Hubysh

01/17/2023, 2:48 PM
Hopefully last question to this thread: I managed to set up and running an agent on ACI. The plan is, that this Agent starts flows in new ACI containers. I am using custom image for agent and the same image I'm goint to use for "worker" containers. Azure container registry is deployed in the same resource group as container instances. I created Service Principal and assigned "Contributor" role on resource group to him. This SP is also stored as ACI Credentials block in Prefect. However, when I starting the flow, Agent Container throws the following error:
Message: The image '...' in container group '...' is not accessible. Please check the image and registry credential.
Have I missed something?
r

Ryan Peden

01/17/2023, 2:59 PM
A couple of options come to mind: • Create a Docker Registry block with your ACR details, then selecting that registry from the drop down box in your ACI block's 'Image Registry` property. • Use the ACI block's 'Identities' field to add a user-assigned managed identity to the container groups the ACI block runs. ◦ Note: I've tested this feature on several types of Azure resources, but not ACR. However according to the docs, it should work.
i

Ivan Hubysh

01/17/2023, 3:17 PM
Option 1 works. So Docker Registry Block may be used to store ACR credentials as well. I can’t thank you enough.
👍 1
r

Ryan Peden

01/17/2023, 3:54 PM
Happy to assist, Ivan. If you have any other questions, feel free to post in #prefect-azure to ensure maximum visibility 😄
👍 1