https://prefect.io logo
Title
a

Alexander Butler

04/17/2022, 8:52 PM
So I have this deployment in Prefect 2.0:
DeploymentSpec(
    flow_location=str((FLOW_DIR / "salesforce.py").absolute()),
    flow_name="elt-salesforce",
    name="sf-production-elt-job",
    schedule=IntervalSchedule(interval=timedelta(hours=1)),
    tags=["pipeline"],
    flow_runner=DockerFlowRunner(image=f"{IMAGE_REPO}/{DBT_IMAGE}:{TAG}", stream_output=True)
)
And it took awhile to come to me as a requirement but I essentially have 2 steps. Step one requires docker image A to do some data pipeline stuff, step 2 needs my custom dbt docker image B to do some transform AFTER step 1. So these two dependent tasks constitute one flow with each step on independent docker images. A flow runner is configured at a deployment level but I dont see a way to configure it at the task or subflow level. Definitely a key req in current state. Please help!
πŸ‘€ 1
Options currently are: https://discourse.prefect.io/t/can-prefect-run-each-task-in-a-different-docker-container/434 Try to follow this route but downgrading to Prefect 1.0 which I would really prefer to stick to 2.0 interface OR Shell steps? Executing
docker run ...
directly
k

Kevin Kho

04/18/2022, 3:22 AM
Will test this tomorrow but I think it should work with Client.create_flow_run_from_deployment
z

Zanie

04/18/2022, 3:38 PM
This is a roadmap item, but not one that we expect to solve until later this year.
πŸš€ 1
You can also use dockerpy to run docker containers from tasks using a Python API, you don’t have to fall back to shell.
πŸ™Œ 1
πŸ‘€ 1
a

Alexander Butler

04/18/2022, 6:40 PM
@Zanie made my day. Sometimes it's the simple things, y'know. I wasn't thinking outside the box, or inside. Or wherever. Just in the Prefect box. docker-py is a perfect shim here.
πŸ‘ 1
a

Anna Geller

04/19/2022, 3:04 PM
It's actually a good analogy because Prefect is more about the arrows than the boxes - check this blog post