Matthew Blau
01/27/2021, 4:32 PMFROM prefecthq/prefect:0.7.1-python3.6
or similar in the Dockerfile, which I have done so. From there I am needing to create a flow that takes this Dockerfile and builds the container and that is where my understanding is weaker and not so clear. I have @task decorations added to various bits of the integration that I am currently attempting to convert over to having Prefect handle the execution of. I am stuck with the how of writing the flow in order to have it work with this docker container. Am I needing a seperate flow.py that takes the Dockerfile of the container, build it, and run the tasks denoted by "@task" within the integration in order for this to be able to be orchestrated by Prefect? If so, how would I write the flow as an example? I feel like my understanding is flawed and would appreciate some help with this. For reference I am running 0.14.1 Thank you all in advance!Sean Talia
01/27/2021, 5:04 PMRunConfig
images for configuring the environment in which you want to run your your flows.
Those images/containers that you're already working with that house all of your dependencies are going to become your RunConfig
images – you need to install prefect in those containers (if you want to re-author the Dockerfiles to use one of the prefecthq images as a base, that's fine, but you don't need to, you just need to make sure that prefect
is installed in the image somehow). Once you have a fully-baked image that has prefect + all of your other flow dependencies, you will use that image when you go to define your flow by specifying that the flow's run_config
should be of DockerRun type. This is basically saying to prefect, "hey, when an agent is ready to pick up my flow as work and execute it, it first needs to spin up a container based on this image, and then execute the flow code inside of that container"Matthew Blau
01/27/2021, 5:09 PMSean Talia
01/27/2021, 5:12 PMprefect
doing anything around dockerfiles or issuing docker build/run commands; you'll have your fully-baked image ready to go, and you'll feed the name of that image to the DockerRun
config so that prefect knows that the flow's body (i.e. all the tasks it's going to end up executing) is to be run inside a container spun up off that imagefrom prefect import Flow
from prefect.run_configs import DockerRun
import numpy as np
@task
def task_1():
arr = np.array([1, 2, 3, 4, 5])
print(arr)
with Flow(
name="my-example-flow",
run_config=DockerRun(
image="numpy-image",
env={'ENV_VAR1': 'value1'}
)
) as flow:
task_1()
where numpy-image
is a docker image that you've built that has numpy installed in it (along with prefect); a prefect agent will know that when it's time to pick up the flow run, it needs to spin up a container using the numpy-image
baseMatthew Blau
01/27/2021, 5:23 PMSean Talia
01/27/2021, 5:31 PMMatthew Blau
01/27/2021, 5:39 PMSean Talia
01/27/2021, 5:46 PM@task
that in turn executes the program, but in that case you're just sort of turning prefect into a scheduling tool and not getting all the niceties around retry logic, dependency management, parallelization, etc. that you otherwise wouldMatthew Blau
01/27/2021, 5:50 PMSean Talia
01/27/2021, 9:17 PM