Toby Rahloff
09/23/2022, 2:20 PMaddress
param removed from the code snippet for readability).
It seems like resetting the Prefect database fixes this issue. Could the tracking of tasks overload the DB? Is this edge-case known and are there workarounds for it?Jeff Hale
09/23/2022, 2:21 PMToby Rahloff
09/23/2022, 2:26 PMpip freeze
and poetry pyproject.toml belowJeff Hale
09/23/2022, 2:28 PMToby Rahloff
09/23/2022, 2:30 PMfailed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
bug reported by Oscar Björhn for me (Slack ref. )Jeff Hale
09/23/2022, 3:24 PMfailed to parse annotation from 'Name' node: 'NoneType' object has no attribute 'resolve'
should be forthcoming very soon.
The improvements in 2.4.1 (2.4.2 presumably after the fix above) might eliminate the issue totally for you, but I think that’s correct.Toby Rahloff
09/23/2022, 3:37 PMZanie
09/23/2022, 3:51 PMprefect version
?Toby Rahloff
09/23/2022, 4:17 PM❯ prefect --version
2.4.1
Jeff Hale
09/23/2022, 4:21 PMprefect version
From the cli, no dashes, should give a bunch more output.Toby Rahloff
09/23/2022, 4:21 PMtime.sleep
), Ray outperformed Dask by far due to the memory congestion that happens on the Dask Head Node when managing over ~1000 tasks.
It also outperformed the ConcurrentTaskRunner in our tests.
"We’re working with the Ray team" that is amazing to hear! Is there a way to follow this effort? Please also let me know if we can support this in any way❯ prefect version
Version: 2.4.1
API version: 0.8.0
Python version: 3.8.10
Git commit: e941523d
Built: Thu, Sep 22, 2022 12:26 PM
OS/Arch: linux/x86_64
Profile: default
Server type: ephemeral
Server:
Database: sqlite
SQLite version: 3.31.1
Zanie
09/23/2022, 4:22 PMprefect orion start
) and use Postgres once you’re at scale (we have many tasks performing writes in parallel with separate clients which is not SQLite’s forte).Toby Rahloff
09/23/2022, 4:23 PM❯ prefect version
Version: 2.4.1
API version: 0.8.0
Python version: 3.8.10
Git commit: e941523d
Built: Thu, Sep 22, 2022 12:26 PM
OS/Arch: linux/x86_64
Profile: dev
Server type: cloud
Zanie
09/23/2022, 4:23 PMCan you share under which circumstances Ray is the slowest TaskRunner?That’s interesting to hear! I just noticed it was slow when I was building it out.
Toby Rahloff
09/23/2022, 4:24 PMZanie
09/23/2022, 4:25 PMToby Rahloff
09/23/2022, 4:28 PMZanie
09/23/2022, 4:30 PMToby Rahloff
09/23/2022, 4:39 PMZanie
09/23/2022, 4:40 PMasyncio.gather
or anyio.TaskGroup
)Toby Rahloff
09/23/2022, 4:48 PMZanie
09/23/2022, 4:52 PMsleep(1)
they’ll have more spread out requests for orchestration 🙂 it’s weird that the flow run is freezing though, there’s no case where that should be happeningToby Rahloff
09/26/2022, 7:29 AM