Esdras Lopes Nani
11/01/2022, 4:29 PMNico Neumann
11/01/2022, 4:34 PMAnthony Head
11/01/2022, 4:43 PMChris Reuter
11/01/2022, 4:51 PMJamie Blakeman
11/01/2022, 5:08 PMnathaniel
Henning Holgersen
11/01/2022, 7:38 PMKartheek PK
11/02/2022, 2:04 AMRyan Peden
11/09/2022, 2:09 PMRasmus Moseholm
11/15/2022, 8:33 AMHenning Holgersen
01/09/2023, 3:58 PMSenthil M.K.
01/10/2023, 9:05 PMRyan Peden
01/14/2023, 9:09 PMGarrick Chu
03/02/2023, 5:10 PMYSF
03/14/2023, 5:03 AMOfir
05/02/2023, 5:20 PM/prefect
prefix.
Otherwise users will get a blank page when visiting /prefect, because the assets won’t be pointing to the right location.
Does anyone have any experience with that?
Should I set PREFECT_UI_SERVE_BASE
env var?
How should the Dockerfile for the agent change?Henning Holgersen
06/05/2023, 8:32 AMmatt_innerspace.io
06/13/2023, 2:36 AMprefect deployment build
(dropping the --infra flag, too many options to keep track of here).matt_innerspace.io
07/07/2023, 8:15 PMazure storage account
- i can't figure out how to properly connect to an azure blobstore. From what I can tell, I need this to be able to deploy code (which exists in bitbucket) to a remote worker pool.
I've entered enough information to make it work, but it fails:
raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '401 Unauthorized' for url '<https://api.prefect.cloud/api/accounts/><my account>/workspaces/<my workspace>/block_types/slug/azure/block_documents/name/azure-block-001?include_secrets=true'
Response: {'detail': 'Invalid authentication credentials'}
For more information check: <https://httpstatuses.com/401>
Is there a document somewhere that describes how to do this?matt_innerspace.io
07/10/2023, 7:59 PMunhealthy
azure aci work-pool?
I can see that it's unhealthy. When I prefect work-pool inspect
it shows defaults, but no real information about the status.
i configured it via prefect deploy
, when it said you don't have any work pools
..Ofir
08/08/2023, 7:08 PMprefect-server
and a prefect-agent
.
What if 90% of the day the prefect-agent
(which is running on a GPU node on the cluster) is idle?
This means it’s underutilized and we waste money for no good reason.
Reference: Airflow provides Kubernetes Executor - on-demand/ad-hoc worker pods.
Since Prefect thought of everything - I’m sure there is either a built-in capability for that or a design pattern for achieving that.
Thanks!Brian Newman
08/18/2023, 1:13 AMprefect config set PREFECT_LOGGING_LEVEL=DEBUG
prefect worker start --pool 'app-service' --name 'app-prefect-azure' --with-healthcheck
Attached is the screenshot from the logs.
If anyone has encountered this issue or has suggestions on how to troubleshoot it, I'd really appreciate the help!
Thanks in advance! 😊Jeremy Hetzel
08/23/2023, 8:07 PMAzure Container Instance:push
worker pool with a custom container image hosted on azurecr.io. Our current Prefect deployments are with GCP Cloud Run Jobs and custom containers hosted on gcr.io (with an agent running on gke autopilot). We are working with data from Microsoft's Planetary Computer (which is vaguely analogous to Google Earth Engine) hosted in azure's West Europe region. I've been tasked with replicating those Cloud Run Jobs with Azure Container Instance Jobs. I'm new to Azure, so my Azure gestalt is lacking.
We created a resource group west-europe
, a container registry at <http://ccorelabs.azurecr.io|ccorelabs.azurecr.io>
, and a managed identity with AcrPull
role at /subscriptions/fee8989c-9995-4501-ba37-0167452a65c7/resourcegroups/west-europe/providers/Microsoft.ManagedIdentity/userAssignedIdentities/prefect-acr
.
I am following these posts:
https://discourse.prefect.io/t/how-to-run-prefect-in-azure-container-instances-aci/2725
https://medium.com/the-prefect-blog/serverless-prefect-flows-with-azure-container-instances-f2442ebc9343
Running a hello-world deployment, I see the flow run crashes with this error log in the prefect dashboard:
Flow run could not be submitted to infrastructure: (DeploymentFailed) At least one resource deployment operation failed. Please list deployment operations for details. Please see <https://aka.ms/arm-deployment-operations> for usage details.
Code: DeploymentFailed
Message: At least one resource deployment operation failed. Please list deployment operations for details. Please see <https://aka.ms/arm-deployment-operations> for usage details.
Target: /subscriptions/fee8989c-9995-4501-ba37-0167452a65c7/resourceGroups/west-europe/providers/Microsoft.Resources/deployments/prefect-get_repo_info-d0a79a93-bf9a-4d73-a071-1bd41a64356e
Exception Details: (InvalidImageRegistryIdentity) The identity in the 'imageRegistryCredentials' of container group 'get_repo_info-d0a79a93-bf9a-4d73-a071-1bd41a64356e' not found in container group identity list.
Code: InvalidImageRegistryIdentity
Message: The identity in the 'imageRegistryCredentials' of container group 'get_repo_info-d0a79a93-bf9a-4d73-a071-1bd41a64356e' not found in container group identity list.
The above is not a tractable, reproducible example. But if anyone with more experience in Azure sees a common cause for the error, I would be grateful.David Yang
08/23/2023, 8:54 PMJeremy Hetzel
08/25/2023, 4:56 PMResponse: {'detail': 'Error creating deployment: <ValidationError: "\'resource_group_name\' is a required property">'}
I created a worker-pool and worker:
prefect work-pool create -t azure-container-instance azure2
prefect worker start -n my-aci-worker -p azure2
And then I have a deployment:
# deployment.py
from os import environ, path
import sys
from prefect.deployments import Deployment
from prefect_azure.container_instance import AzureContainerInstanceJob
from azure.mgmt.containerinstance.models import ContainerGroupIdentity
azure_container_instance_job_block = AzureContainerInstanceJob.load("azure-test")
deployment = Deployment(
name="azure-test",
flow_name="get_repo_info",
version="latest",
description="Test deployment for Azure Container Instance.",
infrastructure=azure_container_instance_job_block,
work_pool_name="azure2",
path="/opt/prefect/flows",
)
if __name__ == "__main__":
deployment.apply()
When I run the deployment, I see an error:
Response: {'detail': 'Error creating deployment: <ValidationError: "\'resource_group_name\' is a required property">'}
For more information check: <https://httpstatuses.com/409>
The AzureContainerInstanceJob.load("azure-test")
has the correct resource_group_name:
(Pdb) azure_container_instance_job_block.resource_group_name
'west-europe'
Do I need to specify a resource_group_name in the azure_container_instance worker? The worker is running on my local laptop. Does it need to be running in an Azure Container Instance instead?Jeremy Hetzel
08/26/2023, 4:21 PMIdentitites
, Subnet IDs
, or DNS Servers
for an Azure Container Instance work pool via the we portal. This appears to block use of images in the Azure Container Registry, because we need to include the ACRManagedIdentity identity
in the identites
list, otherwise the identity is not attached to the container group, and pulling from an azure container registry fails with
Exception Details: (InvalidImageRegistryIdentity) The identity in the 'imageRegistryCredentials' of container group 'get_repo_info-58bac517-8e50-4466-b939-f88722d7c09a' not found in container group identity list.
Code: InvalidImageRegistryIdentity
Message: The identity in the 'imageRegistryCredentials' of container group 'get_repo_info-58bac517-8e50-4466-b939-f88722d7c09a' not found in container group identity list.
03:18:37 PM
prefect.flow_runs.worker
Jeremy Hetzel
08/30/2023, 6:48 PMCreate or update Container Group
in Started
status. Just trying to sanity check if we have something misconfigured or if his expected in ACI.Brian Newman
08/30/2023, 11:37 PMChris Burke
08/31/2023, 8:09 PMprefect deployment apply ...
to target the remote instance.
Is this even possible without an API key (open source)? If so, could you please point me to some literature that explains what I need to configure? Thanks!Brian Newman
09/19/2023, 2:55 AM