https://prefect.io logo
#prefect-community
Title
# prefect-community
b

Bruno Nunes

04/03/2022, 6:54 PM
Hello, I went through the steps in this tutorial to deploy prefect orion 2.0b2 in k8s (if you provide a namespace other than default the service and deployment are created but the role and the rolebinding are only applied if namespace = default). When I apply the manifest the service and deployment seem to be done correctly but it doesn't create a work queue as described here. Furthermore, the logs show an unusual IP used for the api URL:
Configure Prefect to communicate with the server with:
prefect config set PREFECT_API_URL=<http://0.0.0.0:4200/api>
Check out the dashboard at <http://0.0.0.0:4200>
12:51:20.774 | WARNING | prefect.agent - No work queue found named 'kubernetes'
I've edited the service to be type LoadBalanced and used the external IP to set the PREFECT_API_URL
prefect config set PREFECT_API_URL=<http://xx.xx.xx.xx:4200/api>
I've updated the sqlalchemy as suggested in here and created a new work-queue called kubernetes. I've created a new storage pointing to my azure blob storage and created the new deployment and run it. Everything finishes without errors but I don't see any activity in my cluster nor nothing being created in the UI. Can you give me some guidance on what I might be missing?
a

Anna Geller

04/04/2022, 11:25 AM
@Bruno Nunes I can try to reproduce the error using a local API later today, but for now you can set it up reliably using Cloud 2.0 - here is a working example: create work-queue polling for tag "local" (to indicate a local machine for the agent process and a local Kubernetes cluster):
Copy code
prefect work-queue create local -t local
and an agent for this queue:
Copy code
prefect agent start QUEUE_ID
and in a separate terminal window, you can run the commands from the docstring and using this simple flow and deployment example:
Copy code
"""
export AWS_ACCESS_KEY_ID=xxx
export AWS_SECRET_ACCESS_KEY=xxx
export PREFECT_API_URL=<https://api-beta.prefect.io/api/accounts/xxx/workspaces/xxx>
export PREFECT_API_KEY=xxx

prefect deployment create flows/kubernetes_flow.py
prefect deployment run kubernetes-flow/dev
kubectl get pods --watch
"""
from prefect import flow, task, get_run_logger
from prefect.deployments import DeploymentSpec
from prefect.flow_runners import KubernetesFlowRunner
import platform
import os


@task
def say_hi():
    logger = get_run_logger()
    <http://logger.info|logger.info>("Hello world!")


@task
def print_platform_info():
    logger = get_run_logger()
    <http://logger.info|logger.info>(
        "Platform information: IP = %s, Python = %s, Platform type = %s, OS Version = %s",
        platform.node(),
        platform.python_version(),
        platform.platform(),
        platform.version(),
    )


@flow
def kubernetes_flow():
    hi = say_hi()
    print_platform_info(wait_for=[hi])


DeploymentSpec(
    name="dev",
    flow=kubernetes_flow,
    tags=["local"],
    flow_runner=KubernetesFlowRunner(
        env=dict(
            AWS_ACCESS_KEY_ID=os.environ.get("AWS_ACCESS_KEY_ID"),
            AWS_SECRET_ACCESS_KEY=os.environ.get("AWS_SECRET_ACCESS_KEY"),
            PREFECT_API_URL=os.environ.get("PREFECT_API_URL"),
            PREFECT_API_KEY=os.environ.get("PREFECT_API_KEY"),
        ),
    ),
)
replace all "xxx" with your credentials - this assumes S3 storage, thus the AWS credentials
b

Bruno Nunes

04/04/2022, 6:43 PM
@Anna Geller thanks for the reply. I don't have AWS credentials nor a Cloud2.0 subscription. I'm trying to evaluate prefect2.0 on my current k8s cluster running in Azure. If I start an agent I can see the following error:
20:22:00.508 | INFO    | prefect.agent - Submitting flow run 'dc7a181b-7ad1-487b-85be-6ecdd24210bc'
20:22:00.675 | INFO    | prefect.flow_runner.kubernetes - RUNNING
20:22:02.267 | INFO    | prefect.flow_runner.kubernetes - Flow run 'organic-butterfly' has job settings = {'metadata': {'generateName': 'organic-butterfly', 'namespace': 'default', 'labels': {'io.prefect.flow-run-id': 'dc7a181b-7ad1-487b-85be-6ecdd24210bc', 'io.prefect.flow-run-name': 'organic-butterfly', 'app': 'orion'}}, 'spec': {'template': {'spec': {'restartPolicy': 'Never', 'containers': [{'name': 'job', 'image': 'prefecthq/prefect:2.0b2-python3.8', 'command': ['python', '-m', 'prefect.engine', 'dc7a181b-7ad1-487b-85be-6ecdd24210bc'], 'env': [{'name': 'PREFECT_API_URL', 'value': '<http://orion:4200/api'}]}]}}>, 'backoff_limit': 4}}
20:22:03.057 | INFO    | prefect.agent - Completed submission of flow run 'dc7a181b-7ad1-487b-85be-6ecdd24210bc'
20:22:03.237 | INFO    | prefect.flow_runner.kubernetes - Flow run job 'organic-butterfly9zndk' has status {'active': 1,
'completed_indexes': None,
'completion_time': None,
'conditions': None,
'failed': None,
'ready': None,
'start_time': datetime.datetime(2022, 4, 4, 18, 22, 2, tzinfo=tzutc()),
'succeeded': None,
'uncounted_terminated_pods': None}
20:22:03.238 | INFO    | prefect.flow_runner.kubernetes - Starting watch for pod to start. Job: organic-butterfly9zndk
18:22:05.080 | ERROR   | prefect.engine - Engine execution of flow run 'dc7a181b-7ad1-487b-85be-6ecdd24210bc' exited with unexpected exception
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
yield
File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 101, in connect_tcp
stream: anyio.abc.ByteStream = await anyio.connect_tcp(
File "/usr/local/lib/python3.8/site-packages/anyio/_core/_sockets.py", line 152, in connect_tcp
gai_res = await getaddrinfo(target_host, remote_port, family=family,
File "/usr/local/lib/python3.8/site-packages/anyio/_core/_sockets.py", line 419, in getaddrinfo
gai_res = await get_asynclib().getaddrinfo(encoded_host, port, family=family, type=type,
File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 1570, in getaddrinfo
result = await get_running_loop().getaddrinfo(
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 825, in getaddrinfo
return await self.run_in_executor(
File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
yield
File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
raise exc
File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
response = await connection.handle_async_request(request)
File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection.py", line 86, in handle_async_request
raise exc
File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection.py", line 63, in handle_async_request
stream = await self._connect(request)
File "/usr/local/lib/python3.8/site-packages/httpcore/_async/connection.py", line 111, in _connect
stream = await self._network_backend.connect_tcp(**kwargs)
File "/usr/local/lib/python3.8/site-packages/httpcore/backends/auto.py", line 23, in connect_tcp
return await self._backend.connect_tcp(
File "/usr/local/lib/python3.8/site-packages/httpcore/backends/asyncio.py", line 101, in connect_tcp
stream: anyio.abc.ByteStream = await anyio.connect_tcp(
File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.8/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
raise to_exc(exc)
httpcore.ConnectError: [Errno -2] Name or service not known
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 916, in <module>
enter_flow_run_engine_from_subprocess(flow_run_id)
File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 131, in enter_flow_run_engine_from_subprocess
return anyio.run(retrieve_flow_then_begin_flow_run, flow_run_id)
File "/usr/local/lib/python3.8/site-packages/anyio/_core/_eventloop.py", line 56, in run
return asynclib.run(func, *args, **backend_options)
File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 233, in run
return native_run(wrapper(), debug=debug)
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/usr/local/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 228, in wrapper
return await func(*args)
File "/usr/local/lib/python3.8/site-packages/prefect/client.py", line 82, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/prefect/engine.py", line 191, in retrieve_flow_then_begin_flow_run
flow_run = await client.read_flow_run(flow_run_id)
File "/usr/local/lib/python3.8/site-packages/prefect/client.py", line 1096, in read_flow_run
response = await self._client.get(f"/flow_runs/{flow_run_id}")
File "/usr/local/lib/python3.8/site-packages/prefect/utilities/httpx.py", line 102, in get
return await self.request(
File "/usr/local/lib/python3.8/site-packages/prefect/utilities/httpx.py", line 60, in request
response = await self.send(
File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1593, in send
response = await self._send_handling_auth(
File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1621, in _send_handling_auth
response = await self._send_handling_redirects(
File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1658, in _send_handling_redirects
response = await self._send_single_request(request)
File "/usr/local/lib/python3.8/site-packages/httpx/_client.py", line 1695, in _send_single_request
response = await transport.handle_async_request(request)
File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.8/contextlib.py", line 131, in __exit__
self.gen.throw(type, value, traceback)
File "/usr/local/lib/python3.8/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ConnectError: [Errno -2] Name or service not known
20:22:06.813 | INFO    | prefect.flow_runner.kubernetes - Starting watch for job completion: organic-butterfly9zndk
20:22:11.945 | ERROR   | prefect.flow_runner.kubernetes - Job 'organic-butterfly9zndk' never completed.
a

Anna Geller

04/04/2022, 6:49 PM
you don't need any subscription - sorry, I should have said that. There is a free tier, and you don't even need to provide your credit card details, just sign up with your Google or GitHub account. for Azure, you could use Azure storage - option 0
11 Views