Hello I went through the steps in <https orion docs prefect Prefect Community #prefect-community

Hello, I went through the steps in <this tutorial>...

Bruno Nunes

04/03/2022, 6:54 PM

Hello, I went through the steps in this tutorial to deploy prefect orion 2.0b2 in k8s (if you provide a namespace other than default the service and deployment are created but the role and the rolebinding are only applied if namespace = default). When I apply the manifest the service and deployment seem to be done correctly but it doesn't create a work queue as described here. Furthermore, the logs show an unusual IP used for the api URL:

Configure Prefect to communicate with the server with:

prefect config set PREFECT_API_URL=<http://0.0.0.0:4200/api>

Check out the dashboard at <http://0.0.0.0:4200>

12:51:20.774 | WARNING | prefect.agent - No work queue found named 'kubernetes'

I've edited the service to be type LoadBalanced and used the external IP to set the PREFECT_API_URL

prefect config set PREFECT_API_URL=<http://xx.xx.xx.xx:4200/api>

I've updated the sqlalchemy as suggested in here and created a new work-queue called kubernetes. I've created a new storage pointing to my azure blob storage and created the new deployment and run it. Everything finishes without errors but I don't see any activity in my cluster nor nothing being created in the UI. Can you give me some guidance on what I might be missing?

Anna Geller

04/04/2022, 11:25 AM

@Bruno Nunes I can try to reproduce the error using a local API later today, but for now you can set it up reliably using Cloud 2.0 - here is a working example: create work-queue polling for tag "local" (to indicate a local machine for the agent process and a local Kubernetes cluster):

Copy code

prefect work-queue create local -t local

and an agent for this queue:

Copy code

prefect agent start QUEUE_ID

and in a separate terminal window, you can run the commands from the docstring and using this simple flow and deployment example:

Copy code

"""
export AWS_ACCESS_KEY_ID=xxx
export AWS_SECRET_ACCESS_KEY=xxx
export PREFECT_API_URL=<https://api-beta.prefect.io/api/accounts/xxx/workspaces/xxx>
export PREFECT_API_KEY=xxx

prefect deployment create flows/kubernetes_flow.py
prefect deployment run kubernetes-flow/dev
kubectl get pods --watch
"""
from prefect import flow, task, get_run_logger
from prefect.deployments import DeploymentSpec
from prefect.flow_runners import KubernetesFlowRunner
import platform
import os


@task
def say_hi():
    logger = get_run_logger()
    <http://logger.info|logger.info>("Hello world!")


@task
def print_platform_info():
    logger = get_run_logger()
    <http://logger.info|logger.info>(
        "Platform information: IP = %s, Python = %s, Platform type = %s, OS Version = %s",
        platform.node(),
        platform.python_version(),
        platform.platform(),
        platform.version(),
    )


@flow
def kubernetes_flow():
    hi = say_hi()
    print_platform_info(wait_for=[hi])


DeploymentSpec(
    name="dev",
    flow=kubernetes_flow,
    tags=["local"],
    flow_runner=KubernetesFlowRunner(
        env=dict(
            AWS_ACCESS_KEY_ID=os.environ.get("AWS_ACCESS_KEY_ID"),
            AWS_SECRET_ACCESS_KEY=os.environ.get("AWS_SECRET_ACCESS_KEY"),
            PREFECT_API_URL=os.environ.get("PREFECT_API_URL"),
            PREFECT_API_KEY=os.environ.get("PREFECT_API_KEY"),
        ),
    ),
)

replace all "xxx" with your credentials - this assumes S3 storage, thus the AWS credentials

Bruno Nunes

04/04/2022, 6:43 PM

@Anna Geller thanks for the reply. I don't have AWS credentials nor a Cloud2.0 subscription. I'm trying to evaluate prefect2.0 on my current k8s cluster running in Azure. If I start an agent I can see the following error:

20:22:00.508 | INFO    | prefect.agent - Submitting flow run 'dc7a181b-7ad1-487b-85be-6ecdd24210bc'

20:22:00.675 | INFO    | prefect.flow_runner.kubernetes - RUNNING

20:22:02.267 | INFO    | prefect.flow_runner.kubernetes - Flow run 'organic-butterfly' has job settings = {'metadata': {'generateName': 'organic-butterfly', 'namespace': 'default', 'labels': {'io.prefect.flow-run-id': 'dc7a181b-7ad1-487b-85be-6ecdd24210bc', 'io.prefect.flow-run-name': 'organic-butterfly', 'app': 'orion'}}, 'spec': {'template': {'spec': {'restartPolicy': 'Never', 'containers': [{'name': 'job', 'image': 'prefecthq/prefect:2.0b2-python3.8', 'command': ['python', '-m', 'prefect.engine', 'dc7a181b-7ad1-487b-85be-6ecdd24210bc'], 'env': [{'name': 'PREFECT_API_URL', 'value': '<http://orion:4200/api'}]}]}}>, 'backoff_limit': 4}}

20:22:03.057 | INFO    | prefect.agent - Completed submission of flow run 'dc7a181b-7ad1-487b-85be-6ecdd24210bc'

20:22:03.237 | INFO    | prefect.flow_runner.kubernetes - Flow run job 'organic-butterfly9zndk' has status {'active': 1,

'completed_indexes': None,

'completion_time': None,

'conditions': None,

'failed': None,

'ready': None,

'start_time': datetime.datetime(2022, 4, 4, 18, 22, 2, tzinfo=tzutc()),

'succeeded': None,

'uncounted_terminated_pods': None}

20:22:03.238 | INFO    | prefect.flow_runner.kubernetes - Starting watch for pod to start. Job: organic-butterfly9zndk

18:22:05.080 | ERROR   | prefect.engine - Engine execution of flow run 'dc7a181b-7ad1-487b-85be-6ecdd24210bc' exited with unexpected exception

Traceback (most recent call last):

File "/usr/local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions

yield

File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 101, in connect_tcp

stream: anyio.abc.ByteStream = await anyio.connect_tcp(

File "/usr/local/lib/python3.8/site-packages/anyio/_core/_sockets.py", line 152, in connect_tcp

gai_res = await getaddrinfo(target_host, remote_port, family=family,

File "/usr/local/lib/python3.8/site-packages/anyio/_core/_sockets.py", line 419, in getaddrinfo

gai_res = await get_asynclib().getaddrinfo(encoded_host, port, family=family, type=type,

File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 1570, in getaddrinfo

result = await get_running_loop().getaddrinfo(

File "/usr/local/lib/python3.8/asyncio/base_events.py", line 825, in getaddrinfo

return await self.run_in_executor(

File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run

result = self.fn(*self.args, **self.kwargs)

File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo

for res in _socket.getaddrinfo(host, port, family, type, proto, flags):

socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions

yield

File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request

resp = await self._pool.handle_async_request(req)

File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request

raise exc

File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request

response = await connection.handle_async_request(request)

File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request

raise exc

File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request

stream = await self._connect(request)

File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection.py", line 111, in _connect

stream = await self._network_backend.connect_tcp(**kwargs)

File "/usr/local/lib/python3.8/site-packages/httpcore/backends/auto.py", line 23, in connect_tcp

return await self._backend.connect_tcp(

File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 101, in connect_tcp

stream: anyio.abc.ByteStream = await anyio.connect_tcp(

File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__

self.gen.throw(type, value, traceback)

File "/usr/local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions

raise to_exc(exc)

httpcore.ConnectError: [Errno -2] Name or service not known

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 916, in <module>

enter_flow_run_engine_from_subprocess(flow_run_id)

File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 131, in enter_flow_run_engine_from_subprocess

return anyio.run(retrieve_flow_then_begin_flow_run, flow_run_id)

File "/usr/local/lib/python3.8/site-packages/anyio/_core/_eventloop.py", line 56, in run

return asynclib.run(func, *args, **backend_options)

File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 233, in run

return native_run(wrapper(), debug=debug)

File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run

return loop.run_until_complete(main)

File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete

return future.result()

File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 228, in wrapper

return await func(*args)

File "/usr/local/lib/python3.8/site-packages/prefect/client.py", line 82, in with_injected_client

return await fn(*args, **kwargs)

File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 191, in retrieve_flow_then_begin_flow_run

flow_run = await client.read_flow_run(flow_run_id)

File "/usr/local/lib/python3.8/site-packages/prefect/client.py", line 1096, in read_flow_run

response = await self._client.get(f"/flow_runs/{flow_run_id}")

File "/usr/local/lib/python3.8/site-packages/prefect/utilities/httpx.py", line 102, in get

return await self.request(

File "/usr/local/lib/python3.8/site-packages/prefect/utilities/httpx.py", line 60, in request

response = await self.send(

File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1593, in send

response = await self._send_handling_auth(

File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1621, in _send_handling_auth

response = await self._send_handling_redirects(

File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1658, in _send_handling_redirects

response = await self._send_single_request(request)

File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1695, in _send_single_request

response = await transport.handle_async_request(request)

File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request

resp = await self._pool.handle_async_request(req)

File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__

self.gen.throw(type, value, traceback)

File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions

raise mapped_exc(message) from exc

httpx.ConnectError: [Errno -2] Name or service not known

20:22:06.813 | INFO    | prefect.flow_runner.kubernetes - Starting watch for job completion: organic-butterfly9zndk

20:22:11.945 | ERROR   | prefect.flow_runner.kubernetes - Job 'organic-butterfly9zndk' never completed.

Anna Geller

04/04/2022, 6:49 PM

you don't need any subscription - sorry, I should have said that. There is a free tier, and you don't even need to provide your credit card details, just sign up with your Google or GitHub account. for Azure, you could use Azure storage - option 0

11 Views

Open in Slack

Previous Next