Hello, What would be the best way to run external docker containers as prefect tasks? I have a pipeline/flow that is supposed to run a few "external" docker images on some data (sequentially, where one's output (or part of it) is used as input for the next task). The docker images contain python scripts/modules that are not integrated with prefect. I would like to be able to pull those images from the private registry, and run containers based on images with custom parameters (based on settings defined by the user , that I pass to my flows). The pipeline is meant to be used by users with no prefect knowledge who can just git pull, install everything with pip and run it providing some custom parameters. That's why I would prefer not to go for orion infrastructure blocks and deployment but rather hide all prefect magic under python CLI. The naive approach I was thinking about is to have a custom docker executor class/module (running containers) that methods will be called by prefect tasks, but maybe there is a better way to make it work? I am completely new to prefect, so any advice is much appreciated.

one easy way would be to leverage the DockerContainer block within your flow, task - you could run it from anywhere really once you configure your DockerContainer block from Python code or even from the UI, you can invoke the run method on it to start a given command

Dziekuje! Works great for any image from public registry. Do I understand correctly that I cannot access local docker images?

Hmm this should work as well, if not feel free to open a bug report with a minimal reproducible example on the Prefect repo. You can tag me and I can try to reproduce the issue as well

It does, I must have misunderstood the documentation, once I pointed to local image and blocked pulling everything worked nice and smooth. Thanks a lot. Running dockers with prefect is surprisingly effortless 🙂