Hi, I'm completely new to prefect, I'm currently testing it as a way to run simple pipelines on azure container instances. I'm currently trying to run a snippet from the azure tutorial

from prefect import flow, get_run_logger, task


@task
def log_hello(name="Marvin"):
    logger = get_run_logger()
    logger.info(f"{name} said hello!")
    return name


@flow
def log_hello_flow():
    for name in ["Marvin", "Alice", "Bob"]:
        log_hello(name)


if __name__ == "__main__":
    # log_hello_flow.deploy(
    #     name="local-deployment",
    #     work_pool_name="local-docker-pool",
    #     image="localhost:5000/prefect-test",
    # )

    log_hello_flow.deploy(
        name="my-first-aci-deployment",
        work_pool_name="my-aci-work-pool",
        image="XXX.azurecr.io/prefect-test",
    )

now when I run it with the local docker pool (commented out section) it seems to be working fine. But when I try to run it on ACI I'm getting

Completed submission of flow run '9a62b3a4-c7ff-4d28-96b5-9769694a03d9'
15:25:21.332 | INFO    | prefect.worker.azure-container-instance.azurecontainerworker 4378b7ad-b02b-4c43-b1b9-0b8609a74da9 - AzureContainerInstanceJob 'AzureContainerWorker 4378b7ad-b02b-4c43-b1b9-0b8609a74da9': Container deleted.
15:25:21.374 | INFO    | prefect.flow_runs.worker - Reported flow run '9a62b3a4-c7ff-4d28-96b5-9769694a03d9' as crashed: Flow run infrastructure exited with non-zero status code 1.

The run ends up in a failed state, but I can't find any more info on the issue, neither in prefect nor in azure. On azure activity log I see no failed tasks, the only seeming relevant information there is the following. So the credentials seem fine, as it responds to flow run by creating some container groups but that's it. The image got pushed to ACR as I see it listed there.

did you build the docker image you're using on your own computer?

yes

are you on an m1/2/3/4 mac?

nope

hm so no ARM cpu?

no, Intel

sometimes I see this when there's a mismatch between the build platform and exec platform

sounds plausible but the architecture seems compatbile

yeah that doesn't seem like the issue

I noticed the instance gets created, but I don't see any logs there, and then it just dies and disappears.

maybe it's related to pulling the image from ACR?

so I managed to access container logs

File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 372, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed
An exception occurred.

you're using prefect cloud, right?

nope, so that's probably the issue 😅

yeah, there's no way that ACI is going to be able to talk to your self-hosted server just like that

there went a few hours of my life 😄

I would definitely go make a free prefect cloud account if you're interested in getting code running on azure ASAP

ok, I will check it out

we have this guide for use in conjunction with prefect cloud that should get you going! https://docs.prefect.io/latest/integrations/prefect-azure/aci_worker/

Hi @Paweł Biernat, I'm curious how you access the container logs in Azure? Since the container only exist temporarily I'm now refreshing the logs page in the Azure portal, which is a very tedious task... I'm doing this because not all container logs are visible in the Prefect UI, only the last error message.

if you're using the latest version of the ACI worker, there's an option on the work pool to keep the instance after the flow run is finished

That sounds great, where can I find this option? I can't find it in the base job template

ah it isn't, at least not yet

I was able to inspect the logs even for some time after the containers died with

az container logs --resource-group <rg-name> --name <container-name>

You can list the active containers with

az container list --resource-group <rg-name> --output table

For some reason I still see some of the containers I created a few days ago (all terminated).

@Paweł Biernat Thanks! 🙌