https://prefect.io logo
Join Slack
Powered by
Hello, I am a beginner at prefect pipeline orches...
# ask-community
r
Hello, I am a beginner at prefect pipeline orchestration. This is the first time I've attempted to dockerize my prefect pipeline and register it to local UI server. After submitting my flow, it seems to be stuck with a backlog of "late runs".
Just doing a simple test flow for now:
Copy code
with Flow(
        "Thinknum_ETL",
        schedule=schedule
) as flow:

    token = token_task(
        version=version
    )

    t = print_value(token)

with open('./build_creds.json') as f:
    envs = json.load(f)

storage = Docker(
    image_name='flows-storage',
    image_tag='latest',
    dockerfile='./dockerfiles/thinknum/Dockerfile',
    local_image=True,
    env_vars=envs
)

storage.build()
flow.storage = storage

security = Security(tls_ca_file='certs/myca.pem',
                    tls_client_cert='certs/myca.pem',
                    tls_client_key='certs/mykey.pem',
                    require_encryption=True)
flow.environment.executor = DaskExecutor(address='<IP ADDRESS>', client_kwargs={'security': security})
flow.register(build=False)
j
Hi @Riley Hun Do you have an agent running? If so, what type?
r
Hi @Jenny, Don't think I have an agent running actually.
As an aside, how do I go about testing the dockerized flow without submitting the flow?
j
You'll need an agent to run any flows from the UI in Server - https://docs.prefect.io/orchestration/agents/overview.html
For testing the flow, did you try Josh's suggestion from earlier?
Copy code
from prefect import Flow

my_flow = Flow.load('name_of_flow_OR_abs_path')

my_flow.run()
r
@Jenny I tried it - I named my flow 'Thinknum_ETL', but when I try to load it afterwards, it says it cannot find the flow: 
No such file or directory: '/Users/rihun/.prefect/flows/thinknum-etl.prefect'
j
Just checking that's not a typo - Thinknum_ETL vs thinknum-etl?
Any luck with your agent and late runs?
r
Yup, it's "Thinknum_ETL". Not sure why it's looking for "thinknum-etl". I tried a local agent for now, so after 
flow.register()
, I tried 
flow.run_agent()
. Will try again.
@Jenny It seems to be struggling to connect to the graphql
j
Hmmm... That's strange as you managed to register your flows and have them display in the UI. Are you getting any other error logs?
I'm not sure of your exact set up but it sounds like you're wanting to run flows with Docker. Our tutorial here might be helpful to you: https://docs.prefect.io/orchestration/tutorial/docker.html#persisting-your-flow-with-docker-storage
r
I'm actually taking a step back and now trying to "walk" before I "run". So I just took a simple flow in your docs w/o docker and trying to register to the UI. Still encountering the same problem. My config.toml file is like this:
Copy code
[server]
  
  [server.ui]
    
    graphql_url = "<http://localhost:4200/graphql>"
j
What version of Prefect are you using?
r
Oh whoops never mind. I got that working now. Sorry about that - didn't realize I have to run the flow from the UI.
Will try with docker now
👍 1
@Jenny Seems to work now. The flow did fail though. Kind of hard to tell what the error is based on the logs from the UI. The state message: 
Failed to load and execute Flow's environment: FileNotFoundError(2, 'No such file or directory')
j
Good to hear - you should be able to see more information in the logs page of the UI. I also encourage you to have a look through previous slack messages for that error message - there's some good advice which should help you
👍 1
r
The only log message I get from the UI is this:
Copy code
Submitted for execution: Container ID: 4f32ff2618bbedf51826a325f5517ddd9e610dedff074bb3bc9f7d35a8e1d2e5
Open in Slack
PreviousNext
Powered by