Hello, I am using the

create_app

to start prefect server, but doing so makes my workers fail websocket connections with error 403. Any help on this ?

The issue you're encountering with websocket connections failing with error 403 when using

create_app

to start the Prefect server could be related to several configuration aspects: 1. API URL and Key Configuration: Ensure that the

PREFECT_API_URL

and

PREFECT_API_KEY

are correctly set for your Prefect server instance. This can be done using

prefect cloud login

. More details on managing API keys can be found in Prefect's settings documentation. 2. CSRF Protection: If you are using a reverse proxy (e.g., Nginx or Traefik) to host your Prefect UI, you must configure the server to connect to the API properly, as mismatches can lead to CSRF issues. Check if your

ui.api_url

is set correctly in your configuration. 3. Reverse Proxy Configuration: When hosting the UI behind a reverse proxy, ensure that your external proxy URL is correctly set in the

prefect.toml

file. If not set, the system will default to

api.url

, which might not be correct. 4. Websocket Security: Confirm that any security settings, such as CORS or CSRF protections, are correctly configured to allow websocket connections from your worker nodes. Given your setup, reviewing these configurations should help resolve the 403 error. If the issue persists, you might want to check for any related open issues on Prefect's GitHub repository or reach out to the community for more targeted assistance.

@Marvin I don't use any proxy servers and this runs on kubernetes without any ingress controllers

It seems there are a few potential issues related to websocket connections and Kubernetes deployments that might be causing your problem: 1. Database Communication Timeouts: There have been reports of Prefect server installations experiencing timeouts due to slow database communication, which can affect websocket connections. If you're using a database like PostgreSQL, ensure that it's properly configured and not experiencing timeouts. Issue Details. 2. Version Compatibility: There have been issues with task runs not being registered correctly in some Prefect versions. If you're using a version close to

3.0.0rc19

, consider downgrading or upgrading to a more stable release. Issue Details. 3. Websocket Configuration: Ensure that your Kubernetes setup allows websocket connections to the Prefect API. Although you mentioned not using any proxy servers, verify that your Kubernetes network policies or security groups are not blocking websocket traffic. 4. Resource Allocation: If your Prefect server experiences OOM (Out of Memory) issues, it might affect websocket connections. Consider increasing CPU and memory limits for your Prefect server deployment. If none of these suggestions resolve your issue, ensure your Prefect deployment is up-to-date and consider reaching out to the Prefect community for more targeted assistance.

hi @Sri Sudarsan - why are you using

create_app

instead of

prefect server start

?

Hello @Nate, we have added a logic to add custom authentication around prefect server standalone as we don't have option for the sme

gotcha, thanks for the context!

do you think of a problem because of this ?

well there are things that you'd have to handle yourself if you're not using an official way of running the server like

prefect server start

but if you're implementing custom auth, you likely know that already

yes, I am able to connect to my server from my workers via this client. the only thing that fails is the event clients where it is not able to connect to the websocket endpoints in particular