https://prefect.io logo
Join Slack
Powered by
Hello, We currently have a working deployment of t...
# prefect-kubernetes
g
Hello, We currently have a working deployment of the prefect-server and prefect-worker helm charts our GKE cluster. We're in the process of provisioning new environments and looking to start fresh on 3.x. Are there helm charts available for 3.x? (We are currently testing code changes with the 
3.0.0.rc9
python package)
Looks like 
2024.5.31205053
is packaged with 
3.0.0.rc1
. Is that the latest and is there any potential issues we should be aware of with using that chart with 
3.0.0.rc9
python code? (our workers will be setup to run custom images)
j
hi! we havent released any chart versions that default to the RC's yet
you can override this behavior by setting the image tag in the values files like this:
Copy code
values:
    worker:
      image:
        pullPolicy: Always
        prefectTag: 3.0.0rc9-python3.11-kubernetes
g
Thanks @Jamie Zieziula. Will give that a shot. Do I also need to specify the server tag? And if I'm specifying my own images for the job execution itself, does the worker tag version matter? (my sense is yes b/c it needs the 3.0.0 api?)
Also, should I update to the latest chart version? I'm attempting to use 
2024.5.31205053
atm
j
yeah the latest chart version is always good to be on!
yep server tag should be updated as well!
g
Server is crashing with:
Copy code
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/local/lib/python3.11/site-packages/uvicorn/__main__.py", line 4, in <module>
    uvicorn.main()
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 409, in main
    run(
  File "/usr/local/lib/python3.11/site-packages/uvicorn/main.py", line 575, in run
    server.run()
  File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 62, in run
    return asyncio.run(self.serve(sockets=sockets))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "uvloop/loop.pyx", line 1517, in uvloop.loop.Loop.run_until_complete
  File "/usr/local/lib/python3.11/site-packages/uvicorn/server.py", line 69, in serve
    config.load()
  File "/usr/local/lib/python3.11/site-packages/uvicorn/config.py", line 439, in load
    self.loaded_app = self.loaded_app()
                      ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/server/api/server.py", line 666, in create_app
    ui_app = create_ui_app(ephemeral)
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/server/api/server.py", line 416, in create_ui_app
    create_ui_static_subpath()
  File "/usr/local/lib/python3.11/site-packages/prefect/server/api/server.py", line 391, in create_ui_static_subpath
    os.makedirs(static_dir)
  File "<frozen os>", line 225, in makedirs
OSError: [Errno 30] Read-only file system: '/usr/local/lib/python3.11/site-packages/prefect/server/ui_build'
Server stopped!
Here are my values (included as a subchart):
Copy code
prefect-server:
  server:
    image:
      pullPolicy: Always
      prefectTag: 3.0.0rc9-python3.11-kubernetes  
    replicaCount: 2
Workers also seem to be having an issue:
Copy code
The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/prefect/cli/_utilities.py", line 42, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/cli/_types.py", line 153, in fn
    return asyncio.run(_fn(*args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/cli/worker.py", line 120, in start
    is_paused = await _check_work_pool_paused(work_pool_name)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/cli/worker.py", line 240, in _check_work_pool_paused
    work_pool = await client.read_work_pool(work_pool_name=work_pool_name)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/orchestration.py", line 2613, in read_work_pool
    response = await self._client.get(f"/work_pools/{work_pool_name}")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1801, in get
    return await self.request(
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1574, in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 327, in send
    response = await self._send_with_retry(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/prefect/client/base.py", line 251, in _send_with_retry
    response = await send(request, *send_args, **send_kwargs)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1661, in send
    response = await self._send_handling_auth(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1689, in _send_handling_auth
    response = await self._send_handling_redirects(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1726, in _send_handling_redirects
    response = await self._send_single_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_client.py", line 1763, in _send_single_request
    response = await transport.handle_async_request(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 372, in handle_async_request
    with map_httpcore_exceptions():
  File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/usr/local/lib/python3.11/site-packages/httpx/_transports/default.py", line 86, in map_httpcore_exceptions
    raise mapped_exc(message) from exc
httpx.ConnectError: All connection attempts failed
An exception occurred.
Here are the worker values:
Copy code
prefect-worker:
  worker:
    image:
      pullPolicy: Always
      prefectTag: 3.0.0rc9-python3.11-kubernetes  
    replicaCount: 2
    apiConfig: server
    config:
      workPool: default-compute-pool
      type: kubernetes
    serverApiConfig:
      # Should we set this more dynamically? We may need to figure out how to 
      # play nicely with the helper templates in the subchart to do so
      apiUrl: <http://prefect-server:4200/api>
  role:
    extraPermissions:
      - apiGroups: [""]
        resources: ["events"]
        verbs: ["list", "get", "watch"]
This is what the dependencies look like in our Chart.yaml:
Copy code
dependencies:
  - name: prefect-server
    version: "2024.6.24152434"
    repository: <https://prefecthq.github.io/prefect-helm>
  - name: prefect-worker
    version: "2024.6.24152434"
    repository: <https://prefecthq.github.io/prefect-helm>
Those values (minus the image tags) were working with version "2023.11.16"
38 Views
Open in Slack
PreviousNext
Powered by