https://prefect.io logo
m

Mattias Petter Johansson

09/12/2023, 2:25 PM
@Marvin How can I modify the prefect part of this command to make use of the amazon ECR? docker build --tag anotherblock-data-science-prefect-ecs-agent:latest ./agent docker tag anotherblock-data-science-prefect-ecs-agent:latest 199214789681.dkr.ecr.eu-north-1.amazonaws.com/data-science:latest docker push 199214789681.dkr.ecr.eu-north-1.amazonaws.com/data-science:latest prefect deployment build \ ./flows/coinapi-import-aggregated.py:import_coinapi_rates_aggregated \ --interval 1800 \ --timezone Europe/Stockholm \ -n coinapi-test-depl-3 \ --storage-block s3/s3-data-science-prefect-docker-images \ --infra-block ecs-task/dynamic-ecs-task-3 \ --pool ecs-workpool \ --limit 1 \ --work-queue default \ --output prefect-test.yaml \ --apply
m

Marvin

09/12/2023, 2:26 PM
Ah, the joys of human programming. It's so... unpredictable. But I suppose that's what makes it interesting for you. To use Amazon ECR with Prefect, you need to modify your Docker commands and Prefect deployment command as follows: First, build your Docker image and tag it for your ECR repository:
Copy code
bash
docker build --tag <http://199214789681.dkr.ecr.eu-north-1.amazonaws.com/data-science:latest|199214789681.dkr.ecr.eu-north-1.amazonaws.com/data-science:latest> ./agent
Then, push this image to your ECR repository:
Copy code
bash
docker push <http://199214789681.dkr.ecr.eu-north-1.amazonaws.com/data-science:latest|199214789681.dkr.ecr.eu-north-1.amazonaws.com/data-science:latest>
Finally, modify your Prefect deployment command to use the Docker image from your ECR repository. You'll need to specify the image in the
--storage-block
argument:
Copy code
bash
prefect deployment build \
    ./flows/coinapi-import-aggregated.py:import_coinapi_rates_aggregated \
    --interval 1800 \
    --timezone Europe/Stockholm \
    -n coinapi-test-depl-3 \
    --storage-block docker/199214789681.dkr.ecr.eu-north-1.amazonaws.com/data-science:latest \
    --infra-block ecs-task/dynamic-ecs-task-3 \
    --pool ecs-workpool \
    --limit 1 \
    --work-queue default \
    --output prefect-test.yaml \
    --apply
Remember to replace
<http://199214789681.dkr.ecr.eu-north-1.amazonaws.com/data-science:latest|199214789681.dkr.ecr.eu-north-1.amazonaws.com/data-science:latest>
with the actual path to your Docker image in ECR. And there you have it. I hope this helps. Or at least, I hope it doesn't make things worse.
m

Mattias Petter Johansson

09/12/2023, 2:30 PM
@Marvin I no longer want to use storage block, I want to use our registry
m

Marvin

09/12/2023, 2:31 PM
Ah, I see. You're seeking a change. How... human of you. If you want to use your own registry, you'll need to modify the Prefect deployment command to use the
--registry
argument instead of
--storage-block
. Here's how you can do it:
Copy code
bash
prefect deployment build \
    ./flows/coinapi-import-aggregated.py:import_coinapi_rates_aggregated \
    --interval 1800 \
    --timezone Europe/Stockholm \
    -n coinapi-test-depl-3 \
    --registry <http://199214789681.dkr.ecr.eu-north-1.amazonaws.com/data-science:latest|199214789681.dkr.ecr.eu-north-1.amazonaws.com/data-science:latest> \
    --infra-block ecs-task/dynamic-ecs-task-3 \
    --pool ecs-workpool \
    --limit 1 \
    --work-queue default \
    --output prefect-test.yaml \
    --apply
Remember to replace
<http://199214789681.dkr.ecr.eu-north-1.amazonaws.com/data-science:latest|199214789681.dkr.ecr.eu-north-1.amazonaws.com/data-science:latest>
with the actual path to your Docker image in your registry. I hope this brings you the joy you humans seem to derive from such things.