Has anyone used prefect for running short lived docker containers?

I have not (and I encourage anyone who has to stop by) outside of some testing runs using things like the Docker tasks in the task library https://docs.prefect.io/core/task_library/docker.html or some custom tasks that create containers. What are you looking to do with the container creation? 🙂

I have a crawler written in nodejs that is already beeing run inside a docker contanier

Ah yes so doing something like this with the Docker tasks would accomplish what you want (where each docker function is broken down into it’s own individual task):

from prefect import task, Flow
from prefect.tasks.docker import (
    CreateContainer,
    StartContainer,
    GetContainerLogs,
    WaitOnContainer,
)

create = CreateContainer(image_name="prefecthq/prefect", command="echo 12345")
start = StartContainer()
wait = WaitOnContainer()
logs = GetContainerLogs()


@task
def see_output(out):
    print(out)


with Flow("docker-flow") as flow:
    container_id = create()
    s = start(container_id=container_id)
    w = wait(container_id=container_id)

    l = logs(container_id=container_id)
    l.set_upstream(w)

    see_output(l)

Or optionally you could write a custom task that does all of the docker calls!

And how would I call this other docker tasks from a custom task?

You would probably use the Docker API https://docker-py.readthedocs.io/en/stable/api.html#module-docker.api.container that all of these tasks use (e.g. https://github.com/PrefectHQ/prefect/blob/master/src/prefect/tasks/docker/containers.py#L106)

ah allright, something else I've noticed, the CreateContainer already runs the docker container if a command is provided