Hi @Ryan Peden, When can we expect Updated AzureContainerInstanceJob to pass private Docker registry credentials as a list to be merged, fixing private repos in prefect-azure? Was just about to submit an issue, when I saw you’d already fixed it :)

Hi, I have started looking at running Prefect on Azure Container Images, which looks very promising - especially the ability to assign a user-managed identity is great, I assume it is only a matter of time before this feature becomes available in the UI as well. My question though: is there any timeline for when this feature will stabilize?

Hello there, Are there any task libraries available as OSS for Azure services other than the very limited standard tasks that come with Prefect 2.0?

If anyone is interested in sending Prefect logs to Azure Application Insights, I just posted a short tutorial on how to do it: https://discourse.prefect.io/t/how-to-send-prefect-logs-to-azure-application-insights/2202

Hi All - I'm trying to replicate the Automations feature where: anytime any Flow fails -> log to App Insights (using OpenCensus and with Custom Dimensions) with run details (similar details as the Jinja variables like {{flow.name}}). What are some options and best practices here? cc: @Ryan Peden

Why is this discourse post private: https://discourse.prefect.io/t/how-to-deploy-a-prefect-2-0-agent-to-azure-vm-as-your-execution-layer/1097 ?

How do I run Prefect behind nginx? I have: https://www.my-site.com/app https://www.my-site.com/prefect And I need Prefect to rewrite the URL such that all assets (e.g. css, javascript, etc.) are rewritten with the

/prefect

prefix. Otherwise users will get a blank page when visiting /prefect, because the assets won’t be pointing to the right location. Does anyone have any experience with that? Should I set

PREFECT_UI_SERVE_BASE

env var? How should the Dockerfile for the agent change?

Has anyone been able to run prefect on AKS with either pod identity or workflow identity? The documentation does not touch on how this works when spinning up k8s jobs, but from what I can reason it should be possible to do this by adding a kustomization to the blocks we use. Our goal is to use a single AKS cluster to run agents/workflows in different namespaces with different access. https://learn.microsoft.com/en-us/azure/aks/learn/tutorial-kubernetes-workload-identity

help - i'm trying to flip over to prefect2, and not a prefect expert (coming from a jenkins background), and struggling to follow even the simple example online - https://docs.prefect.io/2.10.13/guides/deployment/aci/ . I have ACIs created. Workpool looks ok. I copy/pasted the code. I'm stuck on the

prefect deployment build

(dropping the --infra flag, too many options to keep track of here).

block setup -

azure storage account

- i can't figure out how to properly connect to an azure blobstore. From what I can tell, I need this to be able to deploy code (which exists in bitbucket) to a remote worker pool. I've entered enough information to make it work, but it fails:

raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '401 Unauthorized' for url '<https://api.prefect.cloud/api/accounts/><my account>/workspaces/<my workspace>/block_types/slug/azure/block_documents/name/azure-block-001?include_secrets=true'
Response: {'detail': 'Invalid authentication credentials'}
For more information check: <https://httpstatuses.com/401>

Is there a document somewhere that describes how to do this?

how to debug an

unhealthy

azure aci work-pool? I can see that it's unhealthy. When I

prefect work-pool inspect

it shows defaults, but no real information about the status. i configured it via

prefect deploy

, when it said

you don't have any work pools

..

Is there a way to have a Prefect GPU agent/worker on (temporary) sleep when it’s not used in Kubernetes? Motivation: let’s assume that the cost of a heavy GPU machine is $2k per day. Let’s also assume that our Prefect deployment runs in AKS (managed Azure Kubernetes) and we have separate pods for

prefect-server

and a

prefect-agent

. What if 90% of the day the

prefect-agent

(which is running on a GPU node on the cluster) is idle? This means it’s underutilized and we waste money for no good reason. Reference: Airflow provides Kubernetes Executor - on-demand/ad-hoc worker pods. Since Prefect thought of everything - I’m sure there is either a built-in capability for that or a design pattern for achieving that. Thanks!

Howdy everyone! 👋 I'm new to using Prefect and I've chosen an Azure App Service with a Linux plan to host the worker. While I've managed to sync it with Prefect Cloud successfully, I'm facing an issue when attempting to run a task. The message I keep seeing is: "Flow run could not be submitted to infrastructure." To start the process, I've been using these CLI commands:

prefect config set PREFECT_LOGGING_LEVEL=DEBUG

prefect worker start --pool 'app-service' --name 'app-prefect-azure' --with-healthcheck

Attached is the screenshot from the logs. If anyone has encountered this issue or has suggestions on how to troubleshoot it, I'd really appreciate the help! Thanks in advance! 😊

I am trying to deploy a Prefect flow with an

Azure Container Instance:push

worker pool with a custom container image hosted on azurecr.io. Our current Prefect deployments are with GCP Cloud Run Jobs and custom containers hosted on gcr.io (with an agent running on gke autopilot). We are working with data from Microsoft's Planetary Computer (which is vaguely analogous to Google Earth Engine) hosted in azure's West Europe region. I've been tasked with replicating those Cloud Run Jobs with Azure Container Instance Jobs. I'm new to Azure, so my Azure gestalt is lacking. We created a resource group

west-europe

, a container registry at

<http://ccorelabs.azurecr.io|ccorelabs.azurecr.io>

, and a managed identity with

AcrPull

role at

/subscriptions/fee8989c-9995-4501-ba37-0167452a65c7/resourcegroups/west-europe/providers/Microsoft.ManagedIdentity/userAssignedIdentities/prefect-acr

. I am following these posts: https://discourse.prefect.io/t/how-to-run-prefect-in-azure-container-instances-aci/2725 https://medium.com/the-prefect-blog/serverless-prefect-flows-with-azure-container-instances-f2442ebc9343 Running a hello-world deployment, I see the flow run crashes with this error log in the prefect dashboard:

Flow run could not be submitted to infrastructure: (DeploymentFailed) At least one resource deployment operation failed. Please list deployment operations for details. Please see <https://aka.ms/arm-deployment-operations> for usage details.
Code: DeploymentFailed
Message: At least one resource deployment operation failed. Please list deployment operations for details. Please see <https://aka.ms/arm-deployment-operations> for usage details.
Target: /subscriptions/fee8989c-9995-4501-ba37-0167452a65c7/resourceGroups/west-europe/providers/Microsoft.Resources/deployments/prefect-get_repo_info-d0a79a93-bf9a-4d73-a071-1bd41a64356e
Exception Details:	(InvalidImageRegistryIdentity) The identity in the 'imageRegistryCredentials' of container group 'get_repo_info-d0a79a93-bf9a-4d73-a071-1bd41a64356e' not found in container group identity list.
	Code: InvalidImageRegistryIdentity
	Message: The identity in the 'imageRegistryCredentials' of container group 'get_repo_info-d0a79a93-bf9a-4d73-a071-1bd41a64356e' not found in container group identity list.

The above is not a tractable, reproducible example. But if anyone with more experience in Azure sees a common cause for the error, I would be grateful.

I got the same issue even the identity has resoure group ownership. I guess that the container group created on the fly doesn't have managed identity assigned with proper permission.

Update: Fixed! Description in the comments. I am seeing this error when deploying an ACI Job deployment:

Response: {'detail': 'Error creating deployment: <ValidationError: "\'resource_group_name\' is a required property">'}

I created a worker-pool and worker:

prefect work-pool create -t azure-container-instance azure2
prefect worker start -n my-aci-worker -p azure2

And then I have a deployment:

# deployment.py

from os import environ, path
import sys
from prefect.deployments import Deployment
from prefect_azure.container_instance import AzureContainerInstanceJob
from azure.mgmt.containerinstance.models import ContainerGroupIdentity

azure_container_instance_job_block = AzureContainerInstanceJob.load("azure-test")

deployment = Deployment(
    name="azure-test",
    flow_name="get_repo_info",
    version="latest",
    description="Test deployment for Azure Container Instance.",
    infrastructure=azure_container_instance_job_block,
    work_pool_name="azure2",
    path="/opt/prefect/flows",
)

if __name__ == "__main__":
    deployment.apply()

When I run the deployment, I see an error:

Response: {'detail': 'Error creating deployment: <ValidationError: "\'resource_group_name\' is a required property">'}
For more information check: <https://httpstatuses.com/409>

The

AzureContainerInstanceJob.load("azure-test")

has the correct resource_group_name:

(Pdb) azure_container_instance_job_block.resource_group_name
'west-europe'

Do I need to specify a resource_group_name in the azure_container_instance worker? The worker is running on my local laptop. Does it need to be running in an Azure Container Instance instead?

It is not possible to set

Identitites

,

Subnet IDs

, or

DNS Servers

for an Azure Container Instance work pool via the we portal. This appears to block use of images in the Azure Container Registry, because we need to include the ACRManagedIdentity

identity

in the

identites

list, otherwise the identity is not attached to the container group, and pulling from an azure container registry fails with

Exception Details:	(InvalidImageRegistryIdentity) The identity in the 'imageRegistryCredentials' of container group 'get_repo_info-58bac517-8e50-4466-b939-f88722d7c09a' not found in container group identity list.
	Code: InvalidImageRegistryIdentity
	Message: The identity in the 'imageRegistryCredentials' of container group 'get_repo_info-58bac517-8e50-4466-b939-f88722d7c09a' not found in container group identity list.
03:18:37 PM
prefect.flow_runs.worker

~Do others have experience deploying images >= 1.5 gb to Azure Container Instances? When we deploy an image of ~600 mb, the container group deploys in a minute or two. When we deploy an image of 1.5 gb, over 20 minutes elapse and the container group

Create or update Container Group

in

Started

status. Just trying to sanity check if we have something misconfigured or if his expected in ACI.

Howdy! Has anyone been able to use an Azure DevOps Pipeline/Release to push to Prefect? This is my current Pipeline YAML file. Attached is the error message when trying to run prefect on line 42.

Hi all, I'm trying to run prefect open source in an Azure kubernetes cluster (along with Airbyte). Currently in the beginning stages of getting this running and trying to deploy a hello world flow to the prefect server in kubernetes. The code for the flow, and everything else are on my machine and I can't figure out how to get

prefect deployment apply ...

to target the remote instance. Is this even possible without an API key (open source)? If so, could you please point me to some literature that explains what I need to configure? Thanks!

Howdy, When using caching on a task. Is there a way to set the storage location to azure blob so that all workers can access it?

Good day everyone! I am trying to get my flow deployed to an ACI - following the steps outlined here: https://docs.prefect.io/2.13.7/guides/deployment/aci/ When running

az container create

I get a successful reply and can see the container in my Azure Console. However, I see no work pools in Prefect. I do see the message attached in the log of the ACI. What am I missing?

Hi, I'm completely new to prefect, I'm currently testing it as a way to run simple pipelines on azure container instances. I'm currently trying to run a snippet from the azure tutorial

from prefect import flow, get_run_logger, task


@task
def log_hello(name="Marvin"):
    logger = get_run_logger()
    logger.info(f"{name} said hello!")
    return name


@flow
def log_hello_flow():
    for name in ["Marvin", "Alice", "Bob"]:
        log_hello(name)


if __name__ == "__main__":
    # log_hello_flow.deploy(
    #     name="local-deployment",
    #     work_pool_name="local-docker-pool",
    #     image="localhost:5000/prefect-test",
    # )

    log_hello_flow.deploy(
        name="my-first-aci-deployment",
        work_pool_name="my-aci-work-pool",
        image="XXX.azurecr.io/prefect-test",
    )

now when I run it with the local docker pool (commented out section) it seems to be working fine. But when I try to run it on ACI I'm getting

Completed submission of flow run '9a62b3a4-c7ff-4d28-96b5-9769694a03d9'
15:25:21.332 | INFO    | prefect.worker.azure-container-instance.azurecontainerworker 4378b7ad-b02b-4c43-b1b9-0b8609a74da9 - AzureContainerInstanceJob 'AzureContainerWorker 4378b7ad-b02b-4c43-b1b9-0b8609a74da9': Container deleted.
15:25:21.374 | INFO    | prefect.flow_runs.worker - Reported flow run '9a62b3a4-c7ff-4d28-96b5-9769694a03d9' as crashed: Flow run infrastructure exited with non-zero status code 1.

The run ends up in a failed state, but I can't find any more info on the issue, neither in prefect nor in azure. On azure activity log I see no failed tasks, the only seeming relevant information there is the following. So the credentials seem fine, as it responds to flow run by creating some container groups but that's it. The image got pushed to ACR as I see it listed there.

Dear prefect folks 🙂 Is there a best practice for how to interact with Azure Event Hubs both for prefect server and prefect cloud? I found Azure's Docs on how to send/receive with Python but no prefect-specfic Azure Event Hub documentation. I also found the archived prefect-azure repository... ... and the new integrations/prefect-azure

I set up a deployment on ACI, and now I can run it with

run_deployment

or via UI. I wanted to change the default instance location from "eastus" to something else with

run_deployment(..., job_variables={"location": "eastus2"})

but the instance is still created at "eastus". Is it possible to change the location on the run level at all? Or can I only change this on the deployment level?

I'm testing the 3.0.0rc9 release and the azure-container-instance base job template looks weird:

"env": {
        "type": "object",
        "title": "Environment Variables",
        "description": "Environment variables to set when starting a flow run.",
        "additionalProperties": {
          "anyOf": [
            {
              "type": "string"
            },
            {
              "type": "null"
            }
          ]
        }
      },
      "name": {
        "anyOf": [
          {
            "type": "string"
          },
          {
            "type": "null"
          }
        ],
        "title": "Name",
        "default": null,
        "description": "Name given to infrastructure created by a worker."

is it supposed to be like that?

In 3.0.0rc9, when starting an azure-container-instance worker I'm getting this error. I'm not sure if this means the worker is still functioning properly or not (I run into base job template issues anyway).

Worker 'AzureContainerWorker 3541d0e6-6cbd-4c20-9588-8f7a8ec53fdf' started!
10:50:07.669 | INFO    | prefect.worker.azure-container-instance.azurecontainerworker 3541d0e6-6cbd-4c20-9588-8f7a8ec53fdf - Work pool 'abc' created.
10:50:08.334 | ERROR   | GlobalEventLoopThread | prefect._internal.concurrency - Service 'EventsWorker' failed with 1 pending items.

here's are the relevant versions

$ poetry show|grep -i prefect
prefect                      3.0.0rc9    Workflow orchestration and managem...
prefect-azure                0.4.0rc2    Prefect integrations with Microsof...

Currently getting the below error when running

prefect deploy

from a GitHub action.

2024-08-28T14:54:39.8060727Z prefect.utilities.dockerutils.PushError: unauthorized: access token has insufficient scopes

Hello community! As part of our efforts to consolidate channels in Prefect's community slack, we're going to be archiving this channel 🪦. Please redirect your messages to #C048K20H0KZ or #CL09KU1K7 instead. We'll be happy to help you out there. Thank you for your understanding 🙇‍♂️!

archived the channel