https://prefect.io
Join Slack
<@ULVA73B9P> how do I pass in a specific docker image to be used in an ecs deployment?
# ask-marvin
t
@Marvin how do I pass in a specific docker image to be used in an ecs deployment?
m
To specify a Docker image for an ECS deployment, you can base it off of one of the provided prefecthq/prefect images, or build your own image. Prefect work pools that use containers refer to the 2-latest image by default. You can specify another image at work pool creation. The work pool image choice can be overridden in individual deployments. Here is an example Dockerfile for building an image based on prefecthq/prefect:2-latest, but with scikit-learn installed:
Copy code
Dockerfile
FROM prefecthq/prefect:2-latest 
RUN pip install scikit-learn
For more details, you can check the Docker guide from Prefect's documentation.
n
image
on the ecs work pool is what you'd set
oh i guess marvin already said, nice job buddy @Marvin
m
Thank you! I'm here to help. If you have any other questions, feel free to ask.
t
ahhhhh ok, I think I might have been thinking about things incorrectly then. can every work pool only spin up containers based on a single image?
I thought I could specify an image per deployment within a work pool
n
> can every work pool only spin up containers based on a single image? nope, marvin's actually right about this > You can specify another image at work pool creation. The work pool image choice can be overridden in individual deployments. you can override the 
image
job variable (where job variables ~= "field on a work pool") for each deployment like this
Copy code
- name: azure-push-check
  entrypoint: src/demo_project/healthcheck.py:healthcheck
  work_pool:
    name: ui-sandbox-worker
    job_variables:
       image: "myrepo/someimage:latest" # or template in a variable / block / env var
docs
t
ok gotcha! whew, I was scared I'd have to go back and reread a ton of docs. I thought I was specifying a new image properly in my deployment
Copy code
work_pool:
      name: my-ecs-pool
      work_queue_name:
      job_variables:
        image: "{{ build-image.image }}"
but it's still trying to pull the boilerplate docker image. let me dig through a couple other things, I think I might have messed up configuring the work pool initially
Copy code
deployments:
  - name: first_flow
    version:
    tags: []
    description:
    entrypoint: flows/flow_elt.py:elt_flow
    parameters: {}
    work_pool:
      name: my-ecs-pool
      work_queue_name:
      job_variables:
        image: "{{ build-image.image }}"
n
what does your build step look like?
t
Copy code
build:
  - prefect.deployments.steps.run_shell_script:
      id: get-commit-hash
      script: git rev-parse --short HEAD
      stream_output: false
  - prefect.deployments.steps.run_shell_script:
      id: ecr-auth-ste
      script: /bin/bash -c 'aws ecr get-login-password --region us-west-2 --profile
        personal | docker login --username AWS --password-stdin XXX.dkr.ecr.us-west-2.amazonaws.com'
      stream_output: true
  - prefect_docker.deployments.steps.build_docker_image:
      id: build_image
      requires: prefect-docker
      image_name: XXX.dkr.ecr.us-west-2.amazonaws.com/dwh
      tag: '{{ get-commit-hash.stdout }}'
      dockerfile: flows/Dockerfile
  - prefect_docker.deployments.steps.push_docker_image:
      requires: prefect-docker
      image_name: XXX.dkr.ecr.us-west-2.amazonaws.com/dwh
      tag: '{{ get-commit-hash.stdout }}'
n
aha
Copy code
id: build_image
seems like it ought to be
Copy code
id: build-image
t
ugh
idk how long I've been staring at this thing
thank you so much
n
🙂
no problem
very nice build step by the way! that's exactly what I think of as the best way to use it
t
thank you! there's some very smart people on this slack, the chat history helped a ton
smart 1
ok! on to some AWS permissions issues now which seems like progress, thank you again!
🦜 1
n
sure thing!
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by