I installed a dask cluster (https://docs.prefect.io/orchestration/recipes/k8s_dask.html#kubernetes-manifests) on our kubernetes cluster and I can run flows on it no problem using flow.run(). However, when I register the flow it tries to run the job as a kubernetes job. I have prefect server deployed and can see the runs, but they all fail with 'module C not found'. I am missing something, why do the flows not execute on the dask cluster?
requests.exceptions.HTTPError: 503 Server Error: Service Unavailable for url: <http://prefect-server-apollo.prefect:4200/graphql>
Bennett Lambert
03/30/2022, 12:51 PM
This was an issue with environment variables inside the launching containers. I needed to remove my HTTP_PROXY environment variable, but keep the HTTPS_PROXY environment variable.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.