https://prefect.io logo
Powered by
Title
s

Sylvain Hazard

11/28/2022, 8:43 AM
Hi there ! Go some weird issue trying out flows locally. I'm running a local orion with 
prefect orion start --port 5000
as well as the very simple flow copied below. Running the flow with 
python log_flow.py
randomly ends up crashing with this error : 
RuntimeError: The connection pool was closed while 2 HTTP requests/responses were still in-flight
. Is this an issue related to the server ? Am I forgetting to await something ? Anything I could do to fix it ?
👀 1
import prefect
from prefect import flow, task, get_run_logger
from utilities import AN_IMPORTED_MESSAGE


@task
def log_task(name):
    logger = get_run_logger()
    <http://logger.info|logger.info>("Hello %s!", name)
    <http://logger.info|logger.info>("Prefect Version = %s 🚀", prefect.__version__)
    logger.debug(AN_IMPORTED_MESSAGE)


@task(description="get_names")
def get_names():
    return range(50)


@flow()
def log_flow():
    names = get_names()

    for name in names:
        log_task.submit(name)


if __name__ == "__main__":
    log_flow()
p

Peyton Runyan

11/28/2022, 12:51 PM
Can you share info about your environment and settings? I just ran this in a loop 100 times on my machine without a crash, but I'm trying to reproduce exactly this error for a bug that I've seen reported.
s

Sylvain Hazard

11/28/2022, 1:07 PM
Sure ! Using Python 3.8.12, Prefect 2.6.9, httpx 0.23.1, httpcore 0.16.1, anyio 3.6.2. Computer is using Ubuntu 20.04.
Any more information could be of use to you ?
I initially had the issue with a more complex flow and managed to narrow the complexity down to this so it definitely looks related to the environment, yeah
:upvote: 1
❤️ 1
p

Peyton Runyan

11/28/2022, 1:10 PM
Which database and prefect API are you using?
s

Sylvain Hazard

11/28/2022, 1:10 PM
Whatever the default is, I suppose
👍 1
p

Peyton Runyan

11/28/2022, 1:11 PM
Awesome - that simplifies things even further 🙂
s

Sylvain Hazard

11/28/2022, 1:11 PM
Started by following the getting started tutorial from the docs
👍 1
Virtualenv was created by pyenv and dependency management is done with Poetry (1.1.11) if that is somehow relevant
:thank-you: 1
p

Peyton Runyan

11/28/2022, 2:31 PM
How frequently are you getting this error?
s

Sylvain Hazard

11/28/2022, 2:46 PM
Hard to say. I'd say it happens about 3 times out of 4 when running the flow, although the number of in-flight requests varies from time to time.
p

Peyton Runyan

11/28/2022, 3:03 PM
Hmmmm. I'm on • ubuntu VM running 22.04 on an M1 Mac using Parallels • python 3.8.15 • httpx 0.23.1 • httpcore 0.16.1 • anyio 3.6.2 • prefect 2.6.9 and can't reproduce.
@Mathijs Carlu Are you experiencing the same issue, can you reproduce with the same minimal example, and what are you running?
s

Sylvain Hazard

11/28/2022, 3:04 PM
Won't have time today but I could try to see if reproducing works in a Docker image ? That might make things easier for you.
p

Peyton Runyan

11/28/2022, 3:05 PM
That would be incredible
s

Sylvain Hazard

11/28/2022, 3:06 PM
I'll keep you posted tomorrow morning most likely (UTC+1)
:thank-you: 1
m

Mathijs Carlu

11/28/2022, 3:15 PM
@Peyton Runyan my problem and setup is shared here. The error I see has been varying. I've seen the one in the shared thread, seen the one in this thread, seen 500 server errors, seen RemoteProtocolErrors (probably some others as well). Mostly, it's one of the mapped tasks that can't manage to report its state to the API so it crashes. The flow continues however, and the task depending on it tries to read its state but can't. I suspect some correlation in all these errors. I could try running the same example, but I don't know when I'll have time for this.
p

Peyton Runyan

11/28/2022, 3:16 PM
Thanks!
@Sylvain Hazard can you upgrading from 
httpcore 0.16.1
to 
0.16.2
and see if that addresses it? Apparently there's an issue with it that has been causing people trouble elsewhere.
Can you also include a full stack trace the next time you get that crash?
s

Sylvain Hazard

11/28/2022, 5:18 PM
I'm home from work already, I'll try it out in the morning. Thanks for digging in !
👍 1
Hey ! It does seem to be working very well, weird that it doesn't happen on your environment. Would you like me to make a PR adding httpcore >= 0.16.2 to the requirements ?
p

Peyton Runyan

11/29/2022, 12:03 PM
That’s great to hear! I’d normally say yes, but I want to check with @Zanie first and see if we just want to avoid 0.16.1 specifically vs forcing 0.16.2 and above.
s

Sylvain Hazard

11/29/2022, 12:25 PM
Right, makes sense. Should I try to make a Docker image that reproduces the issue for further investigation ?
p

Peyton Runyan

11/29/2022, 12:36 PM
Probably not at this point. It looks like the 
connection pool closed with xyz in-flight
is always caused by some sort of upstream error, and this case the error is httpcore. Thanks a ton for working with us on this thought man!
s

Sylvain Hazard

11/29/2022, 12:36 PM
My pleasure, thanks a lot for your time !
❤️ 1
z

Zanie

11/30/2022, 4:01 PM
see if we just want to avoid 0.16.1 specifically vs forcing 0.16.2 and above
Yeah I believe that is better practice.
👍 1
#prefect-communityJoin Slack
Powered by