https://prefect.io logo
Powered by
Title
s

Sean Conroy

12/02/2022, 5:12 PM
Any advice on these errors? Seeing this with Flask + Prefect 2.6.8 running in K8's on GCP. Doesn't seem to affect anything, just gives this weird error in the logs: 
Traceback (most recent call last):
ERROR | asyncio - Task exception was never retrieved
"future: <Task finished name='Task-36' coro=<<async_generator_athrow without __name__>()> exception=KeyError(139836083032272)>"
File "/usr/local/lib/python3.8/dist-packages/prefect/utilities/asyncutils.py", line 263, in on_shutdown yield
GeneratorExit
During handling of the above exception, another exception occurred:
File "/usr/local/lib/python3.8/dist-packages/prefect/utilities/asyncutils.py", line 267, in on_shutdown
    EVENT_LOOP_GC_REFS.pop(key)
KeyError: 139836083032272"
✅ 1
b

Bianca Hoch

12/02/2022, 5:18 PM
Hello Sean, looks like we may have a GitHub issue open that somewhat mirrors the behavior you're seeing here. This is currently being investigated.
s

Sean Conroy

12/02/2022, 5:19 PM
OK good to know - I'll keep my eye on this for a fix - thanks so much!
🙌 1
z

Zanie

12/02/2022, 6:29 PM
Hey! Would you mind throwing the traceback and any additional details into https://github.com/PrefectHQ/prefect/issues/7709
:upvote: 1
:gratitude-thank-you: 1
This is related to our cleanup of async database engines, which is really sketchy because Python does not actually provide a way to perform cleanup on event loop closure.
Are you running your API separately from your flows? ie with 
prefect orion start
?
s

Sean Conroy

12/02/2022, 6:32 PM
Gotcha. Ok so I think unfortunately what I posted above is all the traceback I'm seeing for this... The API is getting created by Flask, I'm not calling 
prefect orion start
anywhere.
import os
import json
from flask import Flask, request
import logging
import traceback
from prefect.orion.schemas.states import StateType

app = Flask(__name__)

#################################################
#
#  Flask routes
#
#################################################
@app.route('/', methods=['POST'])
def default_route_main():

    force_exception = location_success = False
    try:
        # Run flow
        flow_state = my_flow(return_state=True)

        force_exception = flow_state.result()
        flow_success = flow_state.type == (StateType.COMPLETED)
This is the basics of the Flask wrapper
z

Zanie

12/02/2022, 6:44 PM
If you point to a existing API, the database connection won’t be managed inside of Flask which should be more stable
s

Sean Conroy

12/02/2022, 6:45 PM
Gotcha...happy to give that a shot. Sorry for the dumb question - but do you have an example somewhere I could look at for how to do that?
z

Zanie

12/02/2022, 6:48 PM
Basically you run 
prefect orion start
then set the 
PREFECT_API_URL
to localhost in your config.. let me see
https://docs.prefect.io/tutorials/deployments/#run-a-prefect-orion-server
s

Sean Conroy

12/02/2022, 6:49 PM
Thank you!
z

Zanie

12/02/2022, 6:50 PM
For Kubernetes, I think there are better examples
👀 1
Like, you can run this on another pod.
Here’s a helm chart: https://github.com/PrefectHQ/prefect-helm
s

Sean Conroy

12/02/2022, 6:51 PM
Ok yeah - that will help - thanks
z

Zanie

12/02/2022, 6:52 PM
👍 also 
prefect kubernetes manifest orion
2 Views
#prefect-communityJoin Slack
Powered by