@Esdras Lopes Nani has joined the channel

@Nico Neumann has joined the channel

@Anthony Head has joined the channel

@Chris Reuter has joined the channel

@Jamie Blakeman has joined the channel

@nathaniel has joined the channel

@Henning Holgersen has joined the channel

@Kartheek PK has joined the channel

If anyone is interested in using Prefect Orion with a Flexible Postgres Server on Azure, I put together a quick post on how to make it work: https://discourse.prefect.io/t/how-to-use-azure-postgres-flexible-server-with-prefect-orion/1887

Hi @Ryan Peden, When can we expect Updated AzureContainerInstanceJob to pass private Docker registry credentials as a list to be merged, fixing private repos in prefect-azure? Was just about to submit an issue, when I saw you’d already fixed it :)

Hi, I have started looking at running Prefect on Azure Container Images, which looks very promising - especially the ability to assign a user-managed identity is great, I assume it is only a matter of time before this feature becomes available in the UI as well. My question though: is there any timeline for when this feature will stabilize?

Hello there, Are there any task libraries available as OSS for Azure services other than the very limited standard tasks that come with Prefect 2.0?

If anyone is interested in sending Prefect logs to Azure Application Insights, I just posted a short tutorial on how to do it: https://discourse.prefect.io/t/how-to-send-prefect-logs-to-azure-application-insights/2202

Hi All - I'm trying to replicate the Automations feature where: anytime any Flow fails -> log to App Insights (using OpenCensus and with Custom Dimensions) with run details (similar details as the Jinja variables like {{flow.name}}). What are some options and best practices here? cc: @Ryan Peden

Why is this discourse post private: https://discourse.prefect.io/t/how-to-deploy-a-prefect-2-0-agent-to-azure-vm-as-your-execution-layer/1097 ?

How do I run Prefect behind nginx? I have: https://www.my-site.com/app https://www.my-site.com/prefect And I need Prefect to rewrite the URL such that all assets (e.g. css, javascript, etc.) are rewritten with the

/prefect

prefix. Otherwise users will get a blank page when visiting /prefect, because the assets won’t be pointing to the right location. Does anyone have any experience with that? Should I set

PREFECT_UI_SERVE_BASE

env var? How should the Dockerfile for the agent change?

Has anyone been able to run prefect on AKS with either pod identity or workflow identity? The documentation does not touch on how this works when spinning up k8s jobs, but from what I can reason it should be possible to do this by adding a kustomization to the blocks we use. Our goal is to use a single AKS cluster to run agents/workflows in different namespaces with different access. https://learn.microsoft.com/en-us/azure/aks/learn/tutorial-kubernetes-workload-identity

help - i'm trying to flip over to prefect2, and not a prefect expert (coming from a jenkins background), and struggling to follow even the simple example online - https://docs.prefect.io/2.10.13/guides/deployment/aci/ . I have ACIs created. Workpool looks ok. I copy/pasted the code. I'm stuck on the

prefect deployment build

(dropping the --infra flag, too many options to keep track of here).

block setup -

azure storage account

- i can't figure out how to properly connect to an azure blobstore. From what I can tell, I need this to be able to deploy code (which exists in bitbucket) to a remote worker pool. I've entered enough information to make it work, but it fails:

raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '401 Unauthorized' for url '<https://api.prefect.cloud/api/accounts/><my account>/workspaces/<my workspace>/block_types/slug/azure/block_documents/name/azure-block-001?include_secrets=true'
Response: {'detail': 'Invalid authentication credentials'}
For more information check: <https://httpstatuses.com/401>

Is there a document somewhere that describes how to do this?

how to debug an

unhealthy

azure aci work-pool? I can see that it's unhealthy. When I

prefect work-pool inspect

it shows defaults, but no real information about the status. i configured it via

prefect deploy

, when it said

you don't have any work pools

..

Is there a way to have a Prefect GPU agent/worker on (temporary) sleep when it’s not used in Kubernetes? Motivation: let’s assume that the cost of a heavy GPU machine is $2k per day. Let’s also assume that our Prefect deployment runs in AKS (managed Azure Kubernetes) and we have separate pods for

prefect-server

and a

prefect-agent

. What if 90% of the day the

prefect-agent

(which is running on a GPU node on the cluster) is idle? This means it’s underutilized and we waste money for no good reason. Reference: Airflow provides Kubernetes Executor - on-demand/ad-hoc worker pods. Since Prefect thought of everything - I’m sure there is either a built-in capability for that or a design pattern for achieving that. Thanks!

Howdy everyone! 👋 I'm new to using Prefect and I've chosen an Azure App Service with a Linux plan to host the worker. While I've managed to sync it with Prefect Cloud successfully, I'm facing an issue when attempting to run a task. The message I keep seeing is: "Flow run could not be submitted to infrastructure." To start the process, I've been using these CLI commands:

prefect config set PREFECT_LOGGING_LEVEL=DEBUG

prefect worker start --pool 'app-service' --name 'app-prefect-azure' --with-healthcheck

Attached is the screenshot from the logs. If anyone has encountered this issue or has suggestions on how to troubleshoot it, I'd really appreciate the help! Thanks in advance! 😊

I am trying to deploy a Prefect flow with an

Azure Container Instance:push

worker pool with a custom container image hosted on azurecr.io. Our current Prefect deployments are with GCP Cloud Run Jobs and custom containers hosted on gcr.io (with an agent running on gke autopilot). We are working with data from Microsoft's Planetary Computer (which is vaguely analogous to Google Earth Engine) hosted in azure's West Europe region. I've been tasked with replicating those Cloud Run Jobs with Azure Container Instance Jobs. I'm new to Azure, so my Azure gestalt is lacking. We created a resource group

west-europe

, a container registry at

<http://ccorelabs.azurecr.io|ccorelabs.azurecr.io>

, and a managed identity with

AcrPull

role at

/subscriptions/fee8989c-9995-4501-ba37-0167452a65c7/resourcegroups/west-europe/providers/Microsoft.ManagedIdentity/userAssignedIdentities/prefect-acr

. I am following these posts: https://discourse.prefect.io/t/how-to-run-prefect-in-azure-container-instances-aci/2725 https://medium.com/the-prefect-blog/serverless-prefect-flows-with-azure-container-instances-f2442ebc9343 Running a hello-world deployment, I see the flow run crashes with this error log in the prefect dashboard:

Flow run could not be submitted to infrastructure: (DeploymentFailed) At least one resource deployment operation failed. Please list deployment operations for details. Please see <https://aka.ms/arm-deployment-operations> for usage details.
Code: DeploymentFailed
Message: At least one resource deployment operation failed. Please list deployment operations for details. Please see <https://aka.ms/arm-deployment-operations> for usage details.
Target: /subscriptions/fee8989c-9995-4501-ba37-0167452a65c7/resourceGroups/west-europe/providers/Microsoft.Resources/deployments/prefect-get_repo_info-d0a79a93-bf9a-4d73-a071-1bd41a64356e
Exception Details:	(InvalidImageRegistryIdentity) The identity in the 'imageRegistryCredentials' of container group 'get_repo_info-d0a79a93-bf9a-4d73-a071-1bd41a64356e' not found in container group identity list.
	Code: InvalidImageRegistryIdentity
	Message: The identity in the 'imageRegistryCredentials' of container group 'get_repo_info-d0a79a93-bf9a-4d73-a071-1bd41a64356e' not found in container group identity list.

The above is not a tractable, reproducible example. But if anyone with more experience in Azure sees a common cause for the error, I would be grateful.

I got the same issue even the identity has resoure group ownership. I guess that the container group created on the fly doesn't have managed identity assigned with proper permission.

Update: Fixed! Description in the comments. I am seeing this error when deploying an ACI Job deployment:

Response: {'detail': 'Error creating deployment: <ValidationError: "\'resource_group_name\' is a required property">'}

I created a worker-pool and worker:

prefect work-pool create -t azure-container-instance azure2
prefect worker start -n my-aci-worker -p azure2

And then I have a deployment:

# deployment.py

from os import environ, path
import sys
from prefect.deployments import Deployment
from prefect_azure.container_instance import AzureContainerInstanceJob
from azure.mgmt.containerinstance.models import ContainerGroupIdentity

azure_container_instance_job_block = AzureContainerInstanceJob.load("azure-test")

deployment = Deployment(
    name="azure-test",
    flow_name="get_repo_info",
    version="latest",
    description="Test deployment for Azure Container Instance.",
    infrastructure=azure_container_instance_job_block,
    work_pool_name="azure2",
    path="/opt/prefect/flows",
)

if __name__ == "__main__":
    deployment.apply()

When I run the deployment, I see an error:

Response: {'detail': 'Error creating deployment: <ValidationError: "\'resource_group_name\' is a required property">'}
For more information check: <https://httpstatuses.com/409>

The

AzureContainerInstanceJob.load("azure-test")

has the correct resource_group_name:

(Pdb) azure_container_instance_job_block.resource_group_name
'west-europe'

Do I need to specify a resource_group_name in the azure_container_instance worker? The worker is running on my local laptop. Does it need to be running in an Azure Container Instance instead?

It is not possible to set

Identitites

,

Subnet IDs

, or

DNS Servers

for an Azure Container Instance work pool via the we portal. This appears to block use of images in the Azure Container Registry, because we need to include the ACRManagedIdentity

identity

in the

identites

list, otherwise the identity is not attached to the container group, and pulling from an azure container registry fails with

Exception Details:	(InvalidImageRegistryIdentity) The identity in the 'imageRegistryCredentials' of container group 'get_repo_info-58bac517-8e50-4466-b939-f88722d7c09a' not found in container group identity list.
	Code: InvalidImageRegistryIdentity
	Message: The identity in the 'imageRegistryCredentials' of container group 'get_repo_info-58bac517-8e50-4466-b939-f88722d7c09a' not found in container group identity list.
03:18:37 PM
prefect.flow_runs.worker

~Do others have experience deploying images >= 1.5 gb to Azure Container Instances? When we deploy an image of ~600 mb, the container group deploys in a minute or two. When we deploy an image of 1.5 gb, over 20 minutes elapse and the container group

Create or update Container Group

in

Started

status. Just trying to sanity check if we have something misconfigured or if his expected in ACI.

Howdy! Has anyone been able to use an Azure DevOps Pipeline/Release to push to Prefect? This is my current Pipeline YAML file. Attached is the error message when trying to run prefect on line 42.

Hi all, I'm trying to run prefect open source in an Azure kubernetes cluster (along with Airbyte). Currently in the beginning stages of getting this running and trying to deploy a hello world flow to the prefect server in kubernetes. The code for the flow, and everything else are on my machine and I can't figure out how to get

prefect deployment apply ...

to target the remote instance. Is this even possible without an API key (open source)? If so, could you please point me to some literature that explains what I need to configure? Thanks!

Howdy, When using caching on a task. Is there a way to set the storage location to azure blob so that all workers can access it?