I am pushing my code to github then want to build it in a docker image and lastly want to run the prefect pipeline as part of our kubernetes orchestration that we already have set up. When I push my code, I would need to build it and then push it to azure container registry. Then, in kubernetes, I need to run a prefect flow like this one https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/docker_pickle_kubernetes_run_custom_ecr_image.py?

Since you asked for Azure, you should use e.g. this one 😄 https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/azure_kubernetes_run_custom_azure_image.py the one you linked is for AWS EKS and ECR

Right, but why do we need to have a separate task?

what do you mean by separate task? Can you describe what do you try to do? so far I know nothing other than that you are using Azure AKS, ACR and Github 🙂 to give some better recommendation, it would be good to know what do you try to do/orchestrate with Prefect, say a bit more about your use case

Right. I am trying to pull Twilio data everyday via a prefect pipeline.

extract_Twilio = ExtractTwilio()
transform_Twilio = TransformTwilio()
load_Twilio = LoadTwilio()

with Flow(name="TwilioFlow", run_config=UniversalRun(labels=['somethingawesome'])) as flow:
    extracted_Twilio = extract_Twilio(TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN)
    transformed_Twilio = transform_Twilio(extracted_Twilio['results'])
    loaded_Twilio = load_Twilio(transformed_Twilio)

if __name__ == "__main__":
    if ENVIRONMENT == 'local':
        registered_flow_id = flow.register(project_name="Development")
        client.create_flow_run(flow_id=registered_flow_id)

if you are just getting started with deploying orchestration layer for Prefect, this documentation is quite excellent starting point https://docs.prefect.io/orchestration/getting-started/quick-start.html

Oh no I am trying to translate what I did locally, with registering, and running it as part of the prefect cloud.

exactly, the docs I linked explain just that, but if you have any specific questions, happy to help your flow looks good - if you want to use Github storage and AKS cluster with it, you would need to add your Github Secret to Prefect Cloud and add storage and run config to your flow

Hahaha

also from the original question you asked: you can't use both Github and Docker storage. You can either: 1. Use Github storage and build your Docker image on your own and provide the reference to that image on your KubernetesRun run config 2. Use Docker storage and let Prefect build the image + push to ACR automatically for you during flow registration So far I understood you want to have option #1

I actually want option 2

and if for the option #1 you need a bit more help building the Docker image on your own and pushing it to ACR, check out this Dockerfile and those ACR image build commands https://github.com/anna-geller/packaging-prefect-flows/blob/master/azure_docker_build_commands.bash

But I am trying to figure out how I would "trigger" the run in option 2.

why did you mention Github then? 😄

Because that's our version control 😛

you only need to do flow.register() and this triggers the docker build process

is poetry already included in the prefecthq/prefect:1.0.0-python3.8 image?

base image only includes orchestration extras, no custom libraries like poetry https://discourse.prefect.io/t/do-prefect-base-images-include-extras-from-the-task-library-e-g-postgres-for-the-postgres-tasks/437

Sorry apparently I do want to do option 1.. Is there a way to get it to not build the Docker image as part of the storage = Docker? like I want github to build it and push it to ACR, which then I want prefect to pull it from there instead of building itself.

You need to choose, yoiu can't have both 😄 either Github storage with your own custom image that you build yourself or Docker storage where Prefect builds the image on registration If your team generally needs more support in building out your infrastructure and setting up CI/CD pipelines, we have paid support for that: https://discourse.prefect.io/t/how-to-get-in-touch-with-prefect-professional-services/275 But if you want to do it on your own I can help you do that too. To build your image, you would need to: 1. Create a Dockerfile (this is just an example) 2. Run docker build + docker push commands as shown here https://github.com/anna-geller/packaging-prefect-flows/blob/master/azure_docker_build_commands.bash 3. Pass the reference to that image to your KubernetesRun(image="your_image:tag") 4. Specify Github storage with the repo and branch where your code lives 5. Commit your code to Github So it's not that difficult and you can build a CI/CD pipeline to build and push your image automatically upon merge to master branch - if you need some examples, check out https://discourse.prefect.io/tag/ci-cd