After building Docker storage using a custom local...
# prefect-community
l
After building Docker storage using a custom local Dockerfile for multiple flows a la https://docs.prefect.io/cloud/recipes/multi_flow_storage.html#adding-flows-to-storage is it possible to execute a flow in situ locally, running in the container, to closely replicate remote execution?
j
Do you want to run the flow in the context of a cloud run or a local run inside the container to see if it works?
More clearly: do you want the flow to communicate state with Prefect Cloud for this run or no?
l
This would be a local run but executing the
/root/.prefect/my_flow.prefect
within the container to see if it works. In that context, would it still need to connect to Prefect Cloud to get secrets, or would the
Copy code
[cloud]
use_local_secrets = true
and all the configs in my
~/.prefect/config.toml
be honored during this local run?
j
If the
config.toml
inside the container contains your secrets then setting
use_local_secrets
to True would use them otherwise it should be fine grabbing the secrets from cloud if set to false. To run your flow inside that container do something like:
Copy code
from prefect.environments.storage import Docker

flow = Docker().get_flow(flow_location="/root/.prefect/my_flow.prefect")

flow.run()
l
Copy code
Traceback (most recent call last):
  File "./workflows.py", line 221, in <module>
    main()
  File "./workflows.py", line 207, in main
    flow_location=f'/root/.prefect/{flow.name}.prefect'
  File "/home/luke/.pyenv/versions/kepler/lib/python3.7/site-packages/prefect/environments/storage/docker.py", line 214, in get_flow
    with open(flow_location, "rb") as f:
PermissionError: [Errno 13] Permission denied: '/root/.prefect/my_flow.prefect'
j
How are you running the docker container?
docker run -it my_image
? Looks like in the image you build you don’t have root access
l
ah, I was assuming that since flow came from the Docker().get_flow,
flow.run()
would run
docker run -it ...
for you. I ran that and stopped getting the PermissionError. Now it seems to be failing to find a variable I added to my local prefect config. I'm not copying a
config.toml
into the container, so looks like I'll need to figure that out. Thanks for the guidance!