https://prefect.io logo
Join Slack
Powered by
<@ULVA73B9P> - I am having trouble getting my Dock...
# ask-marvin
k
@Marvin - I am having trouble getting my Docker/ECS based deployment to use the docker image I specify. My prefect.yaml looks like so:
Copy code
name: kyle-deployment-test
prefect-version: 2.14.20

build:
  - prefect_docker.deployments.steps.build_docker_image:
      id: build_image
      requires: prefect-docker>=0.6.2
      image_name: 000000000000.dkr.ecr.us-west-2.amazonaws.com/my-repo
      tag: latest
      dockerfile: auto

# push section allows you to manage if and how this project is uploaded to remote locations
push:
  - prefect_docker.deployments.steps.push_docker_image:
      requires: prefect-docker>=0.6.2
      image_name: '{{ build_image.image_name }}'
      tag: '{{ build_image.tag }}'


pull:
  - prefect.deployments.steps.set_working_directory:
      directory: /opt/prefect/test_flow/

# the deployments section allows you to provide configuration for deploying flows
deployments:
  - name: kyle-deployment-test
    version:
    tags: []
    description:
    entrypoint: test_flow.py:flow_test
    parameters: {}
    work_pool:
      name: main-pool-sandbox
      work_queue_name:
      job_variables:
        cluster: prefect-worker-pool-cluster
        image: '{{ build_image.image }}'
    schedules: []
m
Your message was too long by 2 tokens - please shorten it and try again. For reference, here's your message at the allowed limit:
- I am having trouble getting my Docker/ECS based deployment to use the docker image I specify. My prefect.yaml looks like so:
```name: kyle-deployment-test prefect-version: 2.14.20 build: - prefect_docker.deployments.steps.build_docker_image: id: build_image requires: prefect-docker>=0.6.2 image_name: 000000000000.dkr.ecr.us-west-2.amazonaws.com/my-repo tag: latest dockerfile: auto # push section allows you to manage if and how this project is uploaded to remote locations push: - prefect_docker.deployments.steps.push_docker_image: requires: prefect-docker>=0.6.2 image_name: '{{ build_image.image_name }}' tag: '{{ build_image.tag }}' pull: - prefect.deployments.steps.set_working_directory: directory: /opt/prefect/test_flow/ # the deployments section allows you to provide configuration for deploying flows deployments: - name: kyle-deployment-test version: tags: [] description: entrypoint: test_flow.py:flow_test parameters: {} work_pool: name: main-pool-sandbox work_queue_name: job_variables: cluster: prefect-worker-pool-cluster image: '{{ build_image.image }}' schedules:
It seems like you were about to share your 
prefect.yaml
configuration, but it got cut off. Please make sure to provide the content of your 
prefect.yaml
file. This will help me understand the setup and identify any issues that might be occurring. Once you share it, we can look into the Prefect version you're using and any specifics related to Docker and ECS configurations. Let me know if you'd like me to check specific aspects of Prefect's documentation or GitHub issues as well!
k
Copy code
name: kyle-deployment-test
prefect-version: 2.14.20

build:
  - prefect_docker.deployments.steps.build_docker_image:
      id: build_image
      requires: prefect-docker>=0.6.2
      image_name: <http://000000000000.dkr.ecr.us-west-2.amazonaws.com/my-repo|000000000000.dkr.ecr.us-west-2.amazonaws.com/my-repo>
      tag: latest
      dockerfile: auto

# push section allows you to manage if and how this project is uploaded to remote locations
push:
  - prefect_docker.deployments.steps.push_docker_image:
      requires: prefect-docker>=0.6.2
      image_name: '{{ build_image.image_name }}'
      tag: '{{ build_image.tag }}'


pull:
  - prefect.deployments.steps.set_working_directory:
      directory: /opt/prefect/test_flow/

# the deployments section allows you to provide configuration for deploying flows
deployments:
  - name: kyle-deployment-test
    version:
    tags: []
    description:
    entrypoint: test_flow.py:flow_test
    parameters: {}
    work_pool:
      name: main-pool-sandbox
      work_queue_name:
      job_variables:
        cluster: prefect-worker-pool-cluster
        image: '{{ build_image.image }}'
    schedules: []
This is the prefect.yaml, It does not seem like my specification for the image in the deployment is working. The container gets build and pushed, but my flow run task starts with the default/base prefect image.
@Kevin Grismore - mind if I ping you on this again? And if so, any ideas here? I see the image override in the deployment Job variables, it just does not seem to be working (cluster is working)
k
on the ECS task that ran, what image did it say it's using?
k
Screenshot 2024-12-12 at 1.58.01 PM.png
Also tried flow run override
Copy code
prefect deployment run 'flow-test/kyle-deployment-test' --job-variable image=[0000000000000.dkr.ecr.us-west-2.amazonaws.com/experimental-sandbox-prefect-repo:latest](http://0000000000000.dkr.ecr.us-west-2.amazonaws.com/experimental-sandbox-prefect-repo:latest)
Creating flow run for deployment 'flow-test/kyle-deployment-test'...
Created flow run 'robust-wasp'.
└── UUID: 0000000000
└── Parameters: {}
└── Job Variables: {'image': '[0000000000000.dkr.ecr.us-west-2.amazonaws.com/experimental-sandbox-prefect-repo:latest](http://0000000000000.dkr.ecr.us-west-2.amazonaws.com/experimental-sandbox-prefect-repo:latest)'}
└── Scheduled start time: 2024-12-12 14:02:46 MST (now)
└── URL: <https://app.prefect.cloud/account/000000000/workspace/00000000000/runs/flow-run/0000000000>
• Tried setting 
task_definition.containerDefinitions.image
on the deployment, no luck • Tried setting image in the work pool config, no luck
k
are you specifying a task def ARN on the work pool?
or are you letting the worker construct and register tasks?
k
No task def arn on the work pool, worker is allowed to construct and register tasks. I can see that the worker is interacting with the ECS API to create the task definition for each submission, it is just not setting the image on the task def to my custom value.
Screenshot 2024-12-12 at 2.26.50 PM.png
I think I mucked up my base work pool template to no longer have the image variable, my bad yall 🙂
k
ahhh that'll do it
4 Views
Open in Slack
PreviousNext
Powered by