Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
pacc-clearcover-june-12-2023
pacc-may-31-2023
ppcc-may-16-2023
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Title
a
Agnieszka
12/18/2022, 3:19 PMHello,
What would be the best way to run external docker containers as prefect tasks?
I have a pipeline/flow that is supposed to run a few "external" docker images on some data (sequentially, where one's output (or part of it) is used as input for the next task). The docker images contain python scripts/modules that are not integrated with prefect. I would like to be able to pull those images from the private registry, and run containers based on images with custom parameters (based on settings defined by the user , that I pass to my flows). The pipeline is meant to be used by users with no prefect knowledge who can just git pull, install everything with pip and run it providing some custom parameters. That's why I would prefer not to go for orion infrastructure blocks and deployment but rather hide all prefect magic under python CLI.
The naive approach I was thinking about is to have a custom docker executor class/module (running containers) that methods will be called by prefect tasks, but maybe there is a better way to make it work? I am completely new to prefect, so any advice is much appreciated.
✅ 1
:docker_ship: 1
a
Anna Geller
12/19/2022, 1:30 AMone easy way would be to leverage the DockerContainer block within your flow, task - you could run it from anywhere really
once you configure your DockerContainer block from Python code or even from the UI, you can invoke the run method on it to start a given command
🙌 1
let me try to build a small example to make it more concrete
from prefect.infrastructure import DockerContainer
container = DockerContainer(image="busybox", command=["echo", "hello from Prefect"])
container.run()✅ 1
btw Witam serdecznie, milo Cie poznac Agnieszka! 🤗
😀 1
a
Agnieszka
12/19/2022, 10:29 AMDziekuje!
Works great for any image from public registry. Do I understand correctly that I cannot access local docker images?
a
Anna Geller
12/19/2022, 12:56 PMHmm this should work as well, if not feel free to open a bug report with a minimal reproducible example on the Prefect repo. You can tag me and I can try to reproduce the issue as well
a
Agnieszka
12/19/2022, 10:06 PMIt does, I must have misunderstood the documentation, once I pointed to local image and blocked pulling everything worked nice and smooth. Thanks a lot. Running dockers with prefect is surprisingly effortless 🙂
❤️ 1