@Jason Noxon, I've got an update for you! The Azure marketplace Docker agent is not yet updated but for now, we've published a full walkthrough on how you can set up a new VM on Azure and spin up a Docker agent in a robust way, including setting up long-lived permissions for the agent to pull images from ACR and ensuring that the agent starts automatically on VM start (if you ever want to reboot the VM or shut it down for the night).
Check this out and LMK if something is unclear or doesn't work for you:
https://discourse.prefect.io/t/how-to-spin-up-a-docker-agent-on-azure-vm-a-full-walkthrough/407