https://prefect.io logo
Title
r

Riley Hun

08/11/2020, 11:39 PM
I cannot find my dockerized prefect flow in my ".prefect/flows" directory. I'm attempting to run this code, but to no avail because the path to the file doesn't exist:
from prefect import Flow
my_flow = Flow.load('thinknum-etl')
my_flow.run()
c

Chris White

08/11/2020, 11:41 PM
Hi Riley - can you list the contents of
/opt/prefect
in your Docker image for me?
r

Riley Hun

08/11/2020, 11:45 PM
Sorry @Chris White - are you able to provide more context? I'm reading through the documentation to familiarize myself with your ask and came across this [1]. Please excuse my ignorance. [1] https://docs.prefect.io/orchestration/recipes/deployment.html
c

Chris White

08/11/2020, 11:48 PM
No worries; so can you confirm that you are running that code snippet above within your Flow’s docker image?
r

Riley Hun

08/11/2020, 11:52 PM
Oh...I see - I am running that snippet outside of docker.
Maybe it's my poor understanding but I just did something like this from my local terminal:
with Flow("thinknum-etl") as flow:

    token = token_task(
        version=version
    )

    t = print_value(token)

storage = Docker(
    image_name='flows-storage',
    image_tag='latest',
    dockerfile='./dockerfiles/thinknum/Dockerfile',
    local_image=True,
    env_vars=envs
)

flow.storage = storage

security = Security(tls_ca_file='certs/myca.pem',
                    tls_client_cert='certs/myca.pem',
                    tls_client_key='certs/mykey.pem',
                    require_encryption=True)
flow.environment.executor = DaskExecutor(address='<IP Address>', client_kwargs={'security': security})
flow.register(project_name='thinknum-etl', build=False)
Didn't get any any errors after registering the flow. Then I asked how to debug locally without using the UI, and was advised to run this:
from prefect import Flow
my_flow = Flow.load('thinknum-etl')
my_flow.run()
I imagine I am doing this incorrectly.
c

Chris White

08/11/2020, 11:57 PM
So that code snippet you should be running inside your Docker image — Docker storage means a Docker container is built and your flow is stored within the container, not on your local machine
r

Riley Hun

08/12/2020, 12:04 AM
Right - I thought that snippet takes the docker container I submitted and runs it locally so I can better see the error.
c

Chris White

08/12/2020, 12:07 AM
Ah no, it doesn’t do that; you’ll need to run that code within your image to debug
r

Riley Hun

08/12/2020, 12:09 AM
Oh I see - So if I'm now understanding things better, I'll need to have a separate python file with that code snippet and then using docker's CMD command to run it to debug.
c

Chris White

08/12/2020, 12:12 AM
yea, i personally like to do:
docker run -it IMAGE_NAME:TAG bash
and then you have an interactive shell to do anything you want
👍 1
r

Riley Hun

08/12/2020, 12:20 AM
@Chris White So to answer your questions above, after now understanding your ask, the only file in the /opt/prefect folder is "healthcheck.py"
c

Chris White

08/12/2020, 12:25 AM
ok interesting - and what about
~/.prefect/flows
?
r

Riley Hun

08/12/2020, 12:34 AM
@Chris White Can confirm that no flows exist - so ~/.prefect/flows is empty. I guess that it explains the "file doesn't exist" error I was getting from running the flow on the UI.
c

Chris White

08/12/2020, 12:34 AM
FYI I believe that Flows in Docker are stored in
/opt/prefect
now, but either way that does explain it
r

Riley Hun

08/12/2020, 12:37 AM
Okay got it - thanks - will make a mental note. Not really sure why the flow isn't stored in that folder though. Any insight on this? Maybe I'll work backwards and try a simpler flow from the docs first?
c

Chris White

08/12/2020, 12:39 AM
Oh you put
build=False
in your
flow.register
which means your docker image isn’t even being built
r

Riley Hun

08/12/2020, 12:39 AM
Oh I see! Thanks for catching that. Sorry lame question, but when would one use
build=False?
. I only included it because it seemed like it was being commonly used in other code snippets from the slack history.
i

itay livni

08/12/2020, 12:04 PM
@Riley Hun The "only" time to use
build=True
is when you want to update a flow with a new code. Thus creating a new version of the flow.