https://prefect.io logo
Powered by
Title
a

Abhinav Chordia

04/14/2023, 7:43 PM
Hello! I’m looking into setting up a project with the docker recipe where I’m deploying to our internal k8s cluster. Is there a good example of this setup?
c

Chris White

04/17/2023, 7:42 PM
Hi Abhinav! Projects are still in beta so we are still working on filling out our example documentation. Is there a specific issue you're running into?
a

Abhinav Chordia

04/17/2023, 7:44 PM
Just trying to get an example of how to get this working. The only way I’ve been able to deploy to our k8s cluster is by pushing the code to a storage bucket but if we can get away from that by just pushing images, that would be better imo.
c

Chris White

04/17/2023, 7:46 PM
yea you should definitely be able to use pure images with the docker recipe
the most basic way to get setup is to start a k8s worker in your cluster: 
prefect worker start -t kubernetes -p new-k8s-work-pool
and then use 
prefect project init --recipe docker
to setup a dockerized deployment; push the deployment to the 
new-k8s-work-pool
and that should be enough to get your first run
a

Abhinav Chordia

04/17/2023, 7:57 PM
Thanks! Will try this out and see where I get stuck.
🙌 1
what actually triggers a docker image to be built and pushed?
This is my current deployment.yaml 
description: null
entrypoint: null
flow_name: null
name: null
parameters: {}
schedule: null
tags: []
version: null
work_pool:
  job_variables:
    image: '{{ image_name }}'
  name: null
  work_queue_name: null
This is my current prefect.yaml 
# File for configuring project / deployment build, push and pull steps

# Generic metadata about this project
name: hypo
prefect-version: 2.10.3

# build section allows you to manage and build docker images
build:
- prefect_docker.projects.steps.build_docker_image:
    requires: prefect-docker>0.1.0
    image_name: <http://docker.gh.st/hypo|docker.gh.st/hypo>
    tag: dev
    dockerfile: Dockerfile

# push section allows you to manage if and how this project is uploaded to remote locations
push: null

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.projects.steps.set_working_directory:
    directory: /opt/prefect/hypo
I’m trying to deploy a flow like this: 
from hypo.prefect_apps.example.example_flow import call_api
from prefect.deployments import Deployment
from prefect.infrastructure import KubernetesJob, KubernetesImagePullPolicy
from prefect.filesystems import Azure
from prefect.blocks.kubernetes import KubernetesClusterConfig


customizations = [
    {
        "op": "add",
        "path": "/spec/template/spec/containers/0/resources",
        "value": {
            "requests": {
                "cpu": "2000m",
                "memory": "4Gi"
            },
            "limits": {
                "cpu": "4000m",
                "memory": "8Gi",
            }
        },
    }
]
k8s_job = KubernetesJob(
        namespace="prefect",
        image="<http://docker.gh.st/hypo:dev|docker.gh.st/hypo:dev>",
        image_pull_policy=KubernetesImagePullPolicy.ALWAYS,
        finished_job_ttl=300,
        job_watch_timeout_seconds=600,
        pod_watch_timeout_seconds=600,
        service_account_name="prefect-server",
        customizations=customizations,
    )
k8s_job.save("devk8s", overwrite=True)

# az_block = Azure.load("ghprodstorage1") # load a pre-defined block
kubernetes_job_block = KubernetesJob.load("devk8s")

deployment = Deployment.build_from_flow(
    flow=call_api,
    name="example",
    version=1,
    infrastructure=kubernetes_job_block,
    work_queue_name="default",
    path="/user/abhinav",
    # storage=az_block
)

deployment.apply()
c

Chris White

04/17/2023, 8:38 PM
oh i see; you need to use the CLI to deploy new-style deployments. Check out the tutorial here: https://docs.prefect.io/latest/tutorials/projects/
a

Abhinav Chordia

04/17/2023, 8:39 PM
Ah it’s not supported via the sdk?
Let me try out the CLI
Also not sure if I need to add something in the push and pull section of prefect.yaml that’s different to specify that it’s pulling the image? 
# build section allows you to manage and build docker images
build:
- prefect_docker.projects.steps.build_docker_image:
    requires: prefect-docker>0.1.0
    image_name: <http://docker.gh.st/hypo|docker.gh.st/hypo>
    tag: dev
    dockerfile: Dockerfile

# push section allows you to manage if and how this project is uploaded to remote locations
push: null

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect.projects.steps.set_working_directory:
    directory: /opt/prefect/hypo
I wasn’t able to see a clear example of push
c

Chris White

04/17/2023, 8:43 PM
yea, we're trying to move away from the SDK because it creates too many confusing issues (deploying files should happen one layer up, not within a file). If you check out that tutorial it should review a dockerized example. It looks like you correctly have 
"{{ image_name }}"
in your 
deployment.yaml
so if you use the 
prefect deploy
CLI you should be good go to
a

Abhinav Chordia

04/17/2023, 8:53 PM
With the decision that you are moving away from the SDK, how do you believe deployments should work with large teams where multiple people are potentially deploying their own flows within a project? The nice thing about the SDK was that you could setup tooling to ensure people did similar things when deploying their own flows. For example you could abstract that user specific flows land in user paths so people dont clobber over each other.
c

Chris White

04/17/2023, 8:57 PM
Good question - we're working on the multi-deployment / multi-flow story for projects right now in this MVP PR: https://github.com/PrefectHQ/prefect/pull/9217 Essentially, you would have a base 
prefect.yaml
file that contains defaults for the build / push / pull steps and could override these on a per-deployment basis. Also while these core steps provide us with robust building blocks (the explicit nature of the steps is incredibly convenient for extending and debugging compared to the previous blocks-based implementation), there's still a lot we can do to simplify the UX around deployments from this, so an SDK isn't totally out of the question if we can collect enough feedback about the use cases it would solve for.
👀 1
a

Abhinav Chordia

04/17/2023, 9:07 PM
Thanks! That was one of the first things i was trying to figure out was how to setup multiple deployments within a project with Projects. The use case for an SDK imo is heavily tied to supporting multiple engineers building / testing their flows. As we onboard more people internally to use prefect, we want to ensure that there is general tooling that supports at least the following: • Generating unique images for each engineer’s flows while iterating. • Use good defaults but also provide ways to override them as needed.
c

Chris White

04/17/2023, 9:09 PM
That's helpful to know thank you!
a

Abhinav Chordia

04/17/2023, 9:10 PM
btw, i was able to get images built, pushed, and run. Thanks for helping!
🙌 1
c

Chris White

04/17/2023, 9:11 PM
Excellent, anytime!
#prefect-communityJoin Slack
Powered by