Alexander Butler

04/17/2022, 8:52 PM
So I have this deployment in Prefect 2.0:
    flow_location=str((FLOW_DIR / "").absolute()),
    flow_runner=DockerFlowRunner(image=f"{IMAGE_REPO}/{DBT_IMAGE}:{TAG}", stream_output=True)
And it took awhile to come to me as a requirement but I essentially have 2 steps. Step one requires docker image A to do some data pipeline stuff, step 2 needs my custom dbt docker image B to do some transform AFTER step 1. So these two dependent tasks constitute one flow with each step on independent docker images. A flow runner is configured at a deployment level but I dont see a way to configure it at the task or subflow level. Definitely a key req in current state. Please help!
πŸ‘€ 1
Options currently are: Try to follow this route but downgrading to Prefect 1.0 which I would really prefer to stick to 2.0 interface OR Shell steps? Executing
docker run ...

Kevin Kho

04/18/2022, 3:22 AM
Will test this tomorrow but I think it should work with Client.create_flow_run_from_deployment


04/18/2022, 3:38 PM
This is a roadmap item, but not one that we expect to solve until later this year.
πŸš€ 1
You can also use dockerpy to run docker containers from tasks using a Python API, you don’t have to fall back to shell.
πŸ™Œ 1
πŸ‘€ 1

Alexander Butler

04/18/2022, 6:40 PM
@Zanie made my day. Sometimes it's the simple things, y'know. I wasn't thinking outside the box, or inside. Or wherever. Just in the Prefect box. docker-py is a perfect shim here.
πŸ‘ 1

Anna Geller

04/19/2022, 3:04 PM
It's actually a good analogy because Prefect is more about the arrows than the boxes - check this blog post