Maybe someone will need this:
So after some investigation my conclusion is that docker integration with wsl2 is the root cause.
I got some successes after reinstalling wsl, but then it was failing again 😕
strange thing that it sometimes works and sometimes not.
Even stranger, that when I try to run it in wsl2 docker, then it breaks wsl and I cannot even run prefect as "pure python" (without docker, just local python script)
disabling wsl2 docker integration helps, but then I cannot then run docker in wsl 😞
So for now I'll stick to running it in windows docker ...