https://prefect.io logo
Channels
Powered by
n

Nikolaus Landgraf

07/19/2022, 7:15 AM
Good morning! Since the upgrade to 
2.0b8
we have experienced 2 things: • performance of single flows became around 2x slower (we are using the Sequential task runner) and have a lot of database operations in our tasks • Some Flows get interrupted by a 403 forbidden error and do not pick up from where they were left. Is there a way of increasing the lifetime of the token?
a

Anna Geller

07/19/2022, 10:58 AM
can you move the code block to the thread?
#1 We plan to release changes that will likely address that behavior due to changes in handling results and interaction with task runners - I'd suggest trying out version 2.0b9 first once released, and if the performance issue persists in 2.0b9, could you open a GitHub issue for it? #2 I can ask internally but I wonder are you logging in every time you run the flow? Can you share the flow code that caused this error and explain what setup led to it? Opening a separate GitHub issue for that might be helpful as well
n

Nikolaus Landgraf

07/19/2022, 11:59 AM
Copy code
Traceback (most recent call last):

  File "/usr/local/lib/python3.8/dist-packages/prefect/cli/_utilities.py", line 44, in wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/prefect/utilities/asyncio.py", line 193, in wrapper
    return run_async_in_new_loop(async_fn, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/prefect/utilities/asyncio.py", line 140, in run_async_in_new_loop
    return anyio.run(partial(__fn, *args, **kwargs))
  File "/usr/local/lib/python3.8/dist-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/dist-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/usr/local/lib/python3.8/dist-packages/prefect/cli/agent.py", line 65, in start
    await agent.get_and_submit_flow_runs()
  File "/usr/local/lib/python3.8/dist-packages/prefect/agent.py", line 100, in get_and_submit_flow_runs
    work_queue = await self.client.read_work_queue(work_queue_id)
  File "/usr/local/lib/python3.8/dist-packages/prefect/client.py", line 935, in read_work_queue
    response = await self._client.get(f"/work_queues/{id}")
  File "/usr/local/lib/python3.8/dist-packages/httpx/_client.py", line 1751, in get
    return await self.request(
  File "/usr/local/lib/python3.8/dist-packages/httpx/_client.py", line 1527, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
  File "/usr/local/lib/python3.8/dist-packages/prefect/client.py", line 280, in send
    response.raise_for_status()
  File "/usr/local/lib/python3.8/dist-packages/prefect/client.py", line 226, in raise_for_status
    raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '403 Forbidden' for url '<https://api-beta.prefect.io/api/accounts/.../>...
For more information check: <https://httpstatuses.com/403>
An exception occurred.
Successfully logged in and set workspace to '...' in
profile 'default'.
z

Zanie

07/19/2022, 5:04 PM
Some discussion around the performance changes in https://github.com/PrefectHQ/prefect/pull/6004
n

Nikolaus Landgraf

07/21/2022, 3:59 PM
@Anna Geller @Zanie #1 I have installed the 2.0b10 prefect version (there are no release notes yet) today and tested the performance. Unfortunately things did not improve. I have tested the same function once with a prefect flow decorator and once without it and the performance is about 2x worse using the prefect wrapper. This is a huge problem for us as our nightly tasks do not finish in time anymore. Are you aware of this performance degrading? If so, do you have plans of improving / fixing it? Unfortunately, after having setup prefect 2.0. and setup a stable daily flow for around 2 weeks this makes us wonder if we need to find a prefect alternative or not. I would be happy to chat about your plans before making this decision. #2 we do login after every new release. Is it recommended to login before running a flow?
z

Zanie

07/21/2022, 4:02 PM
We have not changed anything related to performance in b10 so I do not expect it to have changed. Can you share the function you’re using?
I don’t have a good example to test with to improve this
n

Nikolaus Landgraf

07/21/2022, 4:05 PM
@Zanie We have seen it change from 
2.0b7
-> 
2.0b8
. Yes I will share it in a bit
z

Zanie

07/21/2022, 4:07 PM
We added a change there that allowed us to interrupt synchronous tasks (which otherwise is not supported by Python).
👍 1
n

Nikolaus Landgraf

07/22/2022, 8:56 AM
sorry, I am having some trouble keeping up with the migrations to newer prefect versions and the breaking changes that come with it 😄
a

Anna Geller

07/22/2022, 10:41 AM
totally understandable @Nikolaus Landgraf! After the GA release, the pace of introducing breaking changes will slow down a bit so 🤞 this won't be such a problem soon
z

Zanie

07/22/2022, 5:06 PM
I’ve introduced a PR to remove interruption when we don’t need it, which should help with your performance.
🙌 1
🙏 1
An issue with a minimum reproducible example would still be really helpful though!
n

Nikolaus Landgraf

07/25/2022, 12:21 PM
@Zanie thanks sounds good. We will wait for the Wednesday release until we proceed due to a similar issue as mentioned here: https://prefect-community.slack.com/archives/CL09KU1K7/p1658537137702589 Will send you the example then
🙌 1
a

Anna Geller

07/25/2022, 3:43 PM
thanks for your flexibility and understanding @Nikolaus Landgraf, much appreciated 🙌
4 Views
Powered by