Any advice on these errors? Seeing this with Flask + Prefect 2.6.8 running in K8's on GCP. Doesn't...

Sean Conroy

12/02/2022, 5:12 PM

Any advice on these errors? Seeing this with Flask + Prefect 2.6.8 running in K8's on GCP. Doesn't seem to affect anything, just gives this weird error in the logs:

Copy code

Traceback (most recent call last):
ERROR | asyncio - Task exception was never retrieved
"future: <Task finished name='Task-36' coro=<<async_generator_athrow without __name__>()> exception=KeyError(139836083032272)>"
File "/usr/local/lib/python3.8/dist-packages/prefect/utilities/asyncutils.py", line 263, in on_shutdown yield
GeneratorExit
During handling of the above exception, another exception occurred:
File "/usr/local/lib/python3.8/dist-packages/prefect/utilities/asyncutils.py", line 267, in on_shutdown
    EVENT_LOOP_GC_REFS.pop(key)
KeyError: 139836083032272"

✅ 1

Bianca Hoch

12/02/2022, 5:18 PM

Hello Sean, looks like we may have a GitHub issue open that somewhat mirrors the behavior you're seeing here. This is currently being investigated.

Sean Conroy

12/02/2022, 5:19 PM

OK good to know - I'll keep my eye on this for a fix - thanks so much!

🙌 1

Zanie

12/02/2022, 6:29 PM

Hey! Would you mind throwing the traceback and any additional details into https://github.com/PrefectHQ/prefect/issues/7709

upvote 1

gratitude thank you 1

Zanie

12/02/2022, 6:30 PM

This is related to our cleanup of async database engines, which is really sketchy because Python does not actually provide a way to perform cleanup on event loop closure.

Zanie

12/02/2022, 6:30 PM

Are you running your API separately from your flows? ie with

prefect orion start

Sean Conroy

12/02/2022, 6:32 PM

Gotcha. Ok so I think unfortunately what I posted above is all the traceback I'm seeing for this... The API is getting created by Flask, I'm not calling

prefect orion start

anywhere.

Sean Conroy

12/02/2022, 6:33 PM

Copy code

import os
import json
from flask import Flask, request
import logging
import traceback
from prefect.orion.schemas.states import StateType

app = Flask(__name__)

#################################################
#
#  Flask routes
#
#################################################
@app.route('/', methods=['POST'])
def default_route_main():

    force_exception = location_success = False
    try:
        # Run flow
        flow_state = my_flow(return_state=True)

        force_exception = flow_state.result()
        flow_success = flow_state.type == (StateType.COMPLETED)

Sean Conroy

12/02/2022, 6:33 PM

This is the basics of the Flask wrapper

Zanie

12/02/2022, 6:44 PM

If you point to a existing API, the database connection won’t be managed inside of Flask which should be more stable

Sean Conroy

12/02/2022, 6:45 PM

Gotcha...happy to give that a shot. Sorry for the dumb question - but do you have an example somewhere I could look at for how to do that?

Zanie

12/02/2022, 6:48 PM

Basically you run

prefect orion start

then set the

PREFECT_API_URL

to localhost in your config.. let me see

Zanie

12/02/2022, 6:48 PM

https://docs.prefect.io/tutorials/deployments/#run-a-prefect-orion-server

Sean Conroy

12/02/2022, 6:49 PM

Thank you!

Zanie

12/02/2022, 6:50 PM

For Kubernetes, I think there are better examples

👀 1

Zanie

12/02/2022, 6:50 PM

Like, you can run this on another pod.

Zanie

12/02/2022, 6:51 PM

Here’s a helm chart: https://github.com/PrefectHQ/prefect-helm

Sean Conroy

12/02/2022, 6:51 PM

Ok yeah - that will help - thanks

Zanie

12/02/2022, 6:52 PM

👍 also

prefect kubernetes manifest orion

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.