Christopher Chong Tau Teng
11/29/2021, 8:09 AMDockerRun
but am faced with issue importing my flow task from another module — when I run my flow registration script below, it complains that src
module is not found. I think I am doing something wrong because I am confused on how DockerRun works.
I have my Python tasks defined as task_test_flow
which reside in a different python module. I then built a docker image of that python module into <http://gcr.io/some-bucket/prefect-flows:v1|gcr.io/some-bucket/prefect-flows:v1>
which I defined inside DockerRun(image='')
here. Then I created the following registration script where I believe I need to import the task (as done so in https://docs.prefect.io/orchestration/flow_config/docker.html#dependency-requirements). But here’s the problem, task_test_flow
is defined in the docker image <http://gcr.io/some-bucket/prefect-flows:v1|gcr.io/some-bucket/prefect-flows:v1>
, and its not located in the same directory as this registration script below…
I guess my question is, as I want to define my task separately from my flow registration script, how can I import my task (that is already built into a docker image) into my flow registration script? Do I even need to do that in the first place?
from datetime import timedelta
from prefect import Flow
from prefect.schedules import IntervalSchedule
from prefect.storage import GCS
from prefect.run_configs import DockerRun
from src.test_flow import task_test_flow
schedule = IntervalSchedule(interval=timedelta(minutes=1))
with Flow("test-flow", schedule) as flow:
task_test_flow()
flow.storage = GCS(bucket="some-bucket")
flow.run_config = DockerRun(image="<http://gcr.io/some-bucket/prefect-flows:v1|gcr.io/some-bucket/prefect-flows:v1>")
flow.register()
Christopher Chong Tau Teng
11/29/2021, 8:56 AMAnna Geller
Anna Geller
Christopher Chong Tau Teng
11/30/2021, 2:38 AM