s

    Sarah Floris

    6 months ago
    I am pushing my code to github then want to build it in a docker image and lastly want to run the prefect pipeline as part of our kubernetes orchestration that we already have set up. When I push my code, I would need to build it and then push it to azure container registry. Then, in kubernetes, I need to run a prefect flow like this one https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/docker_pickle_kubernetes_run_custom_ecr_image.py?
    Anna Geller

    Anna Geller

    6 months ago
    Since you asked for Azure, you should use e.g. this one 😄 https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/azure_kubernetes_run_custom_azure_image.py the one you linked is for AWS EKS and ECR
    s

    Sarah Floris

    6 months ago
    Right, but why do we need to have a separate task?
    Anna Geller

    Anna Geller

    6 months ago
    what do you mean by separate task? Can you describe what do you try to do? so far I know nothing other than that you are using Azure AKS, ACR and Github 🙂 to give some better recommendation, it would be good to know what do you try to do/orchestrate with Prefect, say a bit more about your use case
    s

    Sarah Floris

    6 months ago
    Right. I am trying to pull Twilio data everyday via a prefect pipeline.
    extract_Twilio = ExtractTwilio()
    transform_Twilio = TransformTwilio()
    load_Twilio = LoadTwilio()
    
    with Flow(name="TwilioFlow", run_config=UniversalRun(labels=['somethingawesome'])) as flow:
        extracted_Twilio = extract_Twilio(TWILIO_ACCOUNT_SID, TWILIO_AUTH_TOKEN)
        transformed_Twilio = transform_Twilio(extracted_Twilio['results'])
        loaded_Twilio = load_Twilio(transformed_Twilio)
    
    if __name__ == "__main__":
        if ENVIRONMENT == 'local':
            registered_flow_id = flow.register(project_name="Development")
            client.create_flow_run(flow_id=registered_flow_id)
    Anna Geller

    Anna Geller

    6 months ago
    if you are just getting started with deploying orchestration layer for Prefect, this documentation is quite excellent starting point https://docs.prefect.io/orchestration/getting-started/quick-start.html
    s

    Sarah Floris

    6 months ago
    Oh no I am trying to translate what I did locally, with registering, and running it as part of the prefect cloud.
    Anna Geller

    Anna Geller

    6 months ago
    exactly, the docs I linked explain just that, but if you have any specific questions, happy to help your flow looks good - if you want to use Github storage and AKS cluster with it, you would need to add your Github Secret to Prefect Cloud and add storage and run config to your flow
    For Github and Azure AKS with ACR, you can use this example: https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/github_kubernetes_run_custom_azure_image.py
    with Flow(name="TwilioFlow", 
    storage== GitHub(
        repo="yourgithub_account/your_repo_name",
        path=f"flows/{FLOW_NAME}.py", # just example
        access_token_secret="GITHUB_ACCESS_TOKEN",  
    ), 
    run_config==KubernetesRun(
            image="<http://YOUR_ACR_REGISTRY.azurecr.io/examplepath/flows|YOUR_ACR_REGISTRY.azurecr.io/examplepath/flows>",
            labels=["aks"], # just example agent label
            image_pull_secrets=["aks"],  # see README
        ),
    ) as flow:
    but make sure you first commit your code to Github before running the flow 😅 it may sound silly but you'd be surprised how often one can forget that
    s

    Sarah Floris

    6 months ago
    Hahaha
    Anna Geller

    Anna Geller

    6 months ago
    also from the original question you asked: you can't use both Github and Docker storage. You can either:1. Use Github storage and build your Docker image on your own and provide the reference to that image on your KubernetesRun run config 2. Use Docker storage and let Prefect build the image + push to ACR automatically for you during flow registration So far I understood you want to have option #1
    s

    Sarah Floris

    6 months ago
    I actually want option 2
    Anna Geller

    Anna Geller

    6 months ago
    and if for the option #1 you need a bit more help building the Docker image on your own and pushing it to ACR, check out this Dockerfile and those ACR image build commands https://github.com/anna-geller/packaging-prefect-flows/blob/master/azure_docker_build_commands.bash
    s

    Sarah Floris

    6 months ago
    But I am trying to figure out how I would "trigger" the run in option 2.
    Anna Geller

    Anna Geller

    6 months ago
    why did you mention Github then? 😄
    s

    Sarah Floris

    6 months ago
    Because that's our version control 😛
    Anna Geller

    Anna Geller

    6 months ago
    you only need to do flow.register() and this triggers the docker build process
    for Azure credentials and how to build the Docker storage for Azure ACR, you can also check out this Discourse topic https://discourse.prefect.io/t/how-to-spin-up-a-docker-agent-on-azure-vm-a-full-walkthrough/407#authenticate-with-acr-7
    oh sorry, I reread - you asked how you can "trigger" a flow run rather than how to trigger the registration/build process - you can either:1. attach schedule to your flow so that Prefect automatically triggers your flow runs on schedule 2. trigger a flow run from UI/API or my favorite - using CLI:
    prefect run --name yourflowname --project XXX --watch
    actually (self-promotion warning 😄) this blog post shows you 6 different ways how you can trigger a flow run https://www.prefect.io/blog/how-to-make-your-data-pipelines-more-dynamic-using-parameters-in-prefect/
    s

    Sarah Floris

    6 months ago
    is poetry already included in the prefecthq/prefect:1.0.0-python3.8 image?
    And as for the files that need to be copied over, the path to twilio is ~/prefect-pipeline/Twilio and is self-contained. The actual flow is twilio_flow.py and twilio_pipeline.py. I will copy over just the Twilio directory and it should be self-contained.
    This example has a ton of other directories it is copying over: https://github.com/anna-geller/packaging-prefect-flows/blob/master/Dockerfile
    Why is that?
    Anna Geller

    Anna Geller

    6 months ago
    you can decide what you copy over to your docker image - it's all up to you. This Dockerfile is just an example and there we copy the entire flow_utilities directory becuase we build a python module based on that 🙂
    s

    Sarah Floris

    6 months ago
    Sorry apparently I do want to do option 1.. Is there a way to get it to not build the Docker image as part of the storage = Docker? like I want github to build it and push it to ACR, which then I want prefect to pull it from there instead of building itself.
    Anna Geller

    Anna Geller

    6 months ago
    You need to choose, yoiu can't have both 😄 either Github storage with your own custom image that you build yourself or Docker storage where Prefect builds the image on registration If your team generally needs more support in building out your infrastructure and setting up CI/CD pipelines, we have paid support for that: https://discourse.prefect.io/t/how-to-get-in-touch-with-prefect-professional-services/275 But if you want to do it on your own I can help you do that too. To build your image, you would need to: 1. Create a Dockerfile (this is just an example) 2. Run docker build + docker push commands as shown here https://github.com/anna-geller/packaging-prefect-flows/blob/master/azure_docker_build_commands.bash 3. Pass the reference to that image to your KubernetesRun(image="your_image:tag") 4. Specify Github storage with the repo and branch where your code lives 5. Commit your code to Github So it's not that difficult and you can build a CI/CD pipeline to build and push your image automatically upon merge to master branch - if you need some examples, check out https://discourse.prefect.io/tag/ci-cd