Fina Silva-Santisteban
01/27/2021, 11:55 PMdocker-compose build app
, then docker-compose run app
.
• Prefect Agent runs flows. The Local Agent runs flows with the local setup. This is not feasible for production since we’d need to manually make sure that whichever machine is hosting the flow has all dependencies installed. Better: run flows within a docker container. This way we don’t have to worry about dependencies, since every machine will simply be running the container. To run flows in containers we need to use the Docker Agent.
• The Docker Agent DOES NOT REQUIRE DOCKER STORAGE, but can work with Docker Storage if desired. The Docker Agent can run with a locally available image (image="example/image-name:with-tag"). If Docker Storage is used, you can provide a url (<http://registry_url|registry_url>
)to where your image is hosted, e.g. Docker Hub, and that url is used to get the image.
• I should be able to run the Docker Agent with the image I had previously created with docker-compose. (It doesn’t seem to be able to find the image, even though I specify the absolute path. )Zanie
Fina Silva-Santisteban
01/28/2021, 4:24 PMApp
-->app_logic
-->prefect_flows
Dockerfile
docker-compose.yml
I think I’m not fully understanding if the image refers to the image of the app? But what else could it refer to?Zanie
DockerRun
config you’re setting on your flow, the log showing the image name in your local docker host, and the log of the agent failing to find the image?Fina Silva-Santisteban
01/28/2021, 6:36 PMflow.run_config = DockerRun(
image="/var/lib/docker/concierge_iro_reporting_app:latest"
)
flow.executor = LocalDaskExecutor(scheduler="threads")
flow.register(project_name=project_name)
The registration works, but when I try to run it it’s just stuck in pending mode. When I check the agent it just seems to be waiting for flow runs. What am I missing?Zanie
concierge_iro_reporting_app:latest
Fina Silva-Santisteban
01/28/2021, 6:47 PMZanie
Local(path="my_<http://module.my|module.my>_flow")
or you could set the path for your local storage to somewhere known in your file system and copy it into your image (for this you need to pass stored_as_script=True
)Fina Silva-Santisteban
01/28/2021, 7:50 PMflow.run_config = DockerRun(
image="concierge_iro_reporting_app:latest",
labels=["docker"]
)
flow.storage = Docker(
path="prefect_flows/flows/generate_report_lt_target.py",
stored_as_script=True
)
flow.executor = LocalDaskExecutor(scheduler="threads")
flow.register(project_name=project_name)
It doesn’t seem to be able to locate the file with the flow, see error message in screenshot. When I run my image and check if the files exists it returns true though:
>>> os.path.isfile('prefect_flows/flows/generate_report_lt_target.py')
True
What am I missing?Zanie
Local
storage instead of Docker
storage there and you’ll have to make sure that path is consistent both locally and within your docker image.core
is when you’re using flow.run()
but you’re registering your flow to a backend.Docker
storage with your Dockerfile
as a base file instead of using docker-compose
Fina Silva-Santisteban
01/28/2021, 9:55 PMZanie
env_vars
dict for that part