Alexander Butler
04/17/2022, 8:52 PMDeploymentSpec(
flow_location=str((FLOW_DIR / "salesforce.py").absolute()),
flow_name="elt-salesforce",
name="sf-production-elt-job",
schedule=IntervalSchedule(interval=timedelta(hours=1)),
tags=["pipeline"],
flow_runner=DockerFlowRunner(image=f"{IMAGE_REPO}/{DBT_IMAGE}:{TAG}", stream_output=True)
)
And it took awhile to come to me as a requirement but I essentially have 2 steps.
Step one requires docker image A to do some data pipeline stuff, step 2 needs my custom dbt docker image B to do some transform AFTER step 1. So these two dependent tasks constitute one flow with each step on independent docker images.
A flow runner is configured at a deployment level but I dont see a way to configure it at the task or subflow level. Definitely a key req in current state.
Please help!Alexander Butler
04/17/2022, 8:59 PMdocker run ...
directlyKevin Kho
Zanie
Zanie
Alexander Butler
04/18/2022, 6:40 PMAnna Geller