Channels
#prefect-community
Title
r
Riley Hun
08/11/2020, 6:48 PMHello,
I am a beginner at prefect pipeline orchestration. This is the first time I've attempted to dockerize my prefect pipeline and register it to local UI server.
After submitting my flow, it seems to be stuck with a backlog of "late runs".
Just doing a simple test flow for now:
Copy code
with Flow(
"Thinknum_ETL",
schedule=schedule
) as flow:
token = token_task(
version=version
)
t = print_value(token)
with open('./build_creds.json') as f:
envs = json.load(f)
storage = Docker(
image_name='flows-storage',
image_tag='latest',
dockerfile='./dockerfiles/thinknum/Dockerfile',
local_image=True,
env_vars=envs
)
storage.build()
flow.storage = storage
security = Security(tls_ca_file='certs/myca.pem',
tls_client_cert='certs/myca.pem',
tls_client_key='certs/mykey.pem',
require_encryption=True)
flow.environment.executor = DaskExecutor(address='<IP ADDRESS>', client_kwargs={'security': security})
flow.register(build=False)j
Jenny
08/11/2020, 6:53 PMHi @Riley Hun Do you have an agent running? If so, what type?
r
Riley Hun
08/11/2020, 6:55 PMHi @Jenny,
Don't think I have an agent running actually.
As an aside, how do I go about testing the dockerized flow without submitting the flow?
j
Jenny
08/11/2020, 7:03 PMYou'll need an agent to run any flows from the UI in Server - https://docs.prefect.io/orchestration/agents/overview.html
For testing the flow, did you try Josh's suggestion from earlier?
Copy code
from prefect import Flow
my_flow = Flow.load('name_of_flow_OR_abs_path')
my_flow.run()r
Riley Hun
08/11/2020, 7:48 PM@Jenny I tried it - I named my flow 'Thinknum_ETL', but when I try to load it afterwards, it says it cannot find the flow:
No such file or directory: '/Users/rihun/.prefect/flows/thinknum-etl.prefect'j
Jenny
08/11/2020, 7:52 PMJust checking that's not a typo - Thinknum_ETL vs thinknum-etl?
Any luck with your agent and late runs?
r
Riley Hun
08/11/2020, 8:05 PMYup, it's "Thinknum_ETL". Not sure why it's looking for "thinknum-etl".
I tried a local agent for now, so after
flow.register()flow.run_agent()@Jenny
It seems to be struggling to connect to the graphql
j
Jenny
08/11/2020, 8:41 PMHmmm... That's strange as you managed to register your flows and have them display in the UI. Are you getting any other error logs?
I'm not sure of your exact set up but it sounds like you're wanting to run flows with Docker. Our tutorial here might be helpful to you: https://docs.prefect.io/orchestration/tutorial/docker.html#persisting-your-flow-with-docker-storage
r
Riley Hun
08/11/2020, 8:45 PMI'm actually taking a step back and now trying to "walk" before I "run". So I just took a simple flow in your docs w/o docker and trying to register to the UI. Still encountering the same problem.
My config.toml file is like this:
Copy code
[server]
[server.ui]
graphql_url = "<http://localhost:4200/graphql>"j
Jenny
08/11/2020, 8:47 PMWhat version of Prefect are you using?
r
Riley Hun
08/11/2020, 8:48 PMOh whoops never mind. I got that working now. Sorry about that - didn't realize I have to run the flow from the UI.
Will try with docker now
👍 1
@Jenny Seems to work now. The flow did fail though. Kind of hard to tell what the error is based on the logs from the UI.
The state message:
Failed to load and execute Flow's environment: FileNotFoundError(2, 'No such file or directory')j
Jenny
08/11/2020, 10:50 PMGood to hear - you should be able to see more information in the logs page of the UI. I also encourage you to have a look through previous slack messages for that error message - there's some good advice which should help you
👍 1
r
Riley Hun
08/11/2020, 11:15 PMThe only log message I get from the UI is this:
Copy code
Submitted for execution: Container ID: 4f32ff2618bbedf51826a325f5517ddd9e610dedff074bb3bc9f7d35a8e1d2e5