https://prefect.io logo
r

Riley Hun

08/11/2020, 6:48 PM
Hello, I am a beginner at prefect pipeline orchestration. This is the first time I've attempted to dockerize my prefect pipeline and register it to local UI server. After submitting my flow, it seems to be stuck with a backlog of "late runs".
Just doing a simple test flow for now:
Copy code
with Flow(
        "Thinknum_ETL",
        schedule=schedule
) as flow:

    token = token_task(
        version=version
    )

    t = print_value(token)

with open('./build_creds.json') as f:
    envs = json.load(f)

storage = Docker(
    image_name='flows-storage',
    image_tag='latest',
    dockerfile='./dockerfiles/thinknum/Dockerfile',
    local_image=True,
    env_vars=envs
)

storage.build()
flow.storage = storage

security = Security(tls_ca_file='certs/myca.pem',
                    tls_client_cert='certs/myca.pem',
                    tls_client_key='certs/mykey.pem',
                    require_encryption=True)
flow.environment.executor = DaskExecutor(address='<IP ADDRESS>', client_kwargs={'security': security})
flow.register(build=False)
j

Jenny

08/11/2020, 6:53 PM
Hi @Riley Hun Do you have an agent running? If so, what type?
r

Riley Hun

08/11/2020, 6:55 PM
Hi @Jenny, Don't think I have an agent running actually.
As an aside, how do I go about testing the dockerized flow without submitting the flow?
j

Jenny

08/11/2020, 7:03 PM
You'll need an agent to run any flows from the UI in Server - https://docs.prefect.io/orchestration/agents/overview.html
For testing the flow, did you try Josh's suggestion from earlier?
Copy code
from prefect import Flow

my_flow = Flow.load('name_of_flow_OR_abs_path')

my_flow.run()
r

Riley Hun

08/11/2020, 7:48 PM
@Jenny I tried it - I named my flow 'Thinknum_ETL', but when I try to load it afterwards, it says it cannot find the flow:
No such file or directory: '/Users/rihun/.prefect/flows/thinknum-etl.prefect'
j

Jenny

08/11/2020, 7:52 PM
Just checking that's not a typo - Thinknum_ETL vs thinknum-etl?
Any luck with your agent and late runs?
r

Riley Hun

08/11/2020, 8:05 PM
Yup, it's "Thinknum_ETL". Not sure why it's looking for "thinknum-etl". I tried a local agent for now, so after
flow.register()
, I tried
flow.run_agent()
. Will try again.
@Jenny It seems to be struggling to connect to the graphql
j

Jenny

08/11/2020, 8:41 PM
Hmmm... That's strange as you managed to register your flows and have them display in the UI. Are you getting any other error logs?
I'm not sure of your exact set up but it sounds like you're wanting to run flows with Docker. Our tutorial here might be helpful to you: https://docs.prefect.io/orchestration/tutorial/docker.html#persisting-your-flow-with-docker-storage
r

Riley Hun

08/11/2020, 8:45 PM
I'm actually taking a step back and now trying to "walk" before I "run". So I just took a simple flow in your docs w/o docker and trying to register to the UI. Still encountering the same problem. My config.toml file is like this:
Copy code
[server]
  
  [server.ui]
    
    graphql_url = "<http://localhost:4200/graphql>"
j

Jenny

08/11/2020, 8:47 PM
What version of Prefect are you using?
r

Riley Hun

08/11/2020, 8:48 PM
Oh whoops never mind. I got that working now. Sorry about that - didn't realize I have to run the flow from the UI.
Will try with docker now
👍 1
@Jenny Seems to work now. The flow did fail though. Kind of hard to tell what the error is based on the logs from the UI. The state message:
Failed to load and execute Flow's environment: FileNotFoundError(2, 'No such file or directory')
j

Jenny

08/11/2020, 10:50 PM
Good to hear - you should be able to see more information in the logs page of the UI. I also encourage you to have a look through previous slack messages for that error message - there's some good advice which should help you
👍 1
r

Riley Hun

08/11/2020, 11:15 PM
The only log message I get from the UI is this:
Copy code
Submitted for execution: Container ID: 4f32ff2618bbedf51826a325f5517ddd9e610dedff074bb3bc9f7d35a8e1d2e5