Ian Andres Etnyre Mercader
09/08/2022, 7:59 PMprefect 2.3.1
, prefect-dask: 0.2.0
and got several errors running code similar to this pseudo code :
@task
def dask_context(item):
name="myblock-name"
config = await JSON.load(name)
@flow(task_runner=DaskTaskRunner())
def do_stuff():
with tags("only_4"):
result_list = dask_context.map(data_list)
I got this Exception multiple times after it running for a while (sometimes minutes, sometimes hours):
Encountered exception during execution:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 33, in read
return await self._stream.receive(max_bytes=max_bytes)
File "/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py", line 1265, in receive
await self._protocol.read_event.wait()
File "/usr/local/lib/python3.9/asyncio/locks.py", line 226, in wait
await fut
asyncio.exceptions.CancelledError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 8, in map_exceptions
yield
File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 35, in read
return b""
File "/usr/local/lib/python3.9/site-packages/anyio/_core/_tasks.py", line 118, in __exit__
raise TimeoutError
TimeoutError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 60, in map_httpcore_exceptions
yield
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 253, in handle_async_request
raise exc
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection_pool.py", line 237, in handle_async_request
response = await connection.handle_async_request(request)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/connection.py", line 90, in handle_async_request
return await self._connection.handle_async_request(request)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/http11.py", line 105, in handle_async_request
raise exc
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/http11.py", line 84, in handle_async_request
) = await self._receive_response_headers(**kwargs)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/http11.py", line 148, in _receive_response_headers
event = await self._receive_event(timeout=timeout)
File "/usr/local/lib/python3.9/site-packages/httpcore/_async/http11.py", line 177, in _receive_event
data = await self._network_stream.read(
File "/usr/local/lib/python3.9/site-packages/httpcore/backends/asyncio.py", line 35, in read
return b""
File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.9/site-packages/httpcore/_exceptions.py", line 12, in map_exceptions
raise to_exc(exc)
httpcore.ReadTimeout
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/prefect/engine.py", line 1185, in orchestrate_task_run
result = await task.fn(*args, **kwargs)
File "/tmp/tmpq25d58myprefect/lib/dask_context.py", line 14, in _dask_task
with await get_context(flow_run.parameters[block_name]):
File "/tmp/tmpq25d58myprefect/./lib/config_context.py", line 40, in get_context
config = await JSON.load(name)
File "/usr/local/lib/python3.9/site-packages/prefect/blocks/core.py", line 606, in load
block_document = await client.read_block_document_by_name(
File "/usr/local/lib/python3.9/site-packages/prefect/client.py", line 1268, in read_block_document_by_name
response = await self._client.get(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1751, in get
return await self.request(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1527, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/usr/local/lib/python3.9/site-packages/prefect/client.py", line 257, in send
await super().send(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1614, in send
response = await self._send_handling_auth(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1642, in _send_handling_auth
response = await self._send_handling_redirects(
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1679, in _send_handling_redirects
response = await self._send_single_request(request)
File "/usr/local/lib/python3.9/site-packages/httpx/_client.py", line 1716, in _send_single_request
response = await transport.handle_async_request(request)
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 353, in handle_async_request
resp = await self._pool.handle_async_request(req)
File "/usr/local/lib/python3.9/contextlib.py", line 137, in __exit__
self.gen.throw(typ, value, traceback)
File "/usr/local/lib/python3.9/site-packages/httpx/_transports/default.py", line 77, in map_httpcore_exceptions
raise mapped_exc(message) from exc
httpx.ReadTimeout
I started running this code with a tag concurrency limit of 20 then switched to 4. The more threads the exception was raised faster.
Currently running on a server of 44 cores and 256 GB.Bradley Hurley
09/08/2022, 8:35 PMBrad Clark
09/08/2022, 9:33 PMprefect cloud login
i get
Unable to authenticate with Prefect Cloud. Please ensure your credentials are correct.
Felix Tremblay
09/08/2022, 9:33 PMAaron Goebel
09/08/2022, 10:57 PMYusuf
09/09/2022, 1:06 AMflow = Flow("My imperative flow!")
# define some new tasks
name = Parameter("name")
second_add = add.copy()
# add our tasks to the flow
flow.add_task(add)
flow.add_task(second_add)
flow.add_task(say_hello)
# create non-data dependencies so that `say_hello` waits for `second_add` to finish.
say_hello.set_upstream(second_add, flow=flow)
# create data bindings
add.bind(x=1, y=2, flow=flow)
second_add.bind(x=add, y=100, flow=flow)
say_hello.bind(person=name, flow=flow)
If I wanted to give the add
task a mapped input?Yousef Hosny
09/09/2022, 1:26 AMYaron Levi
09/09/2022, 4:08 AMfrom weekly_bonus_calc import weekly_bonus_calc
from prefect.deployments import Deployment
from prefect.filesystems import GitHub
storage = GitHub(
repository="<https://github.com/yuvital/yuvital-prefect-github.git>"
)
deployment = Deployment.build_from_flow(
flow=weekly_bonus_calc,
name="weekly-bonus-calc",
work_queue_name="work-queue",
storage=storage,
schedule={'cron': '*/5 * * * *', 'timezone': "Asia/Jerusalem"}
)
deployment.apply()
Also, I’ve created a GitHub block via the UI (attaching a screenshot).
Can you point out the correct way to setup this.
Another thing I don’t understand is where are the Git credentials should go since this is obviously a private repo.
Thank you for any help on the issue 🙏ash
09/09/2022, 6:17 AMBal Raj
09/09/2022, 6:27 AMMalavika S Menon
09/09/2022, 6:29 AMdef random_fn():
print("Random")
@flow
def main_flow():
prefect_flow = Flow(name="random_flow", fn=random_fn)
deployment = Deployment.build_from_flow(
flow=prefect_flow,
name=random_deploy,
version=1,
work_queue_name="test-queue",
)
deployment.apply()
I'm trying to build a deployment for a flow that is created dynamically. While the deployment 'random_flow' gets created successfully, when I try to run the same, I get this error.
Flow could not be retrieved from deployment.
Traceback (most recent call last):
File "/home/malavika/.virtualenvs/lib/python3.8/site-packages/prefect/engine.py", line 256, in retrieve_flow_then_begin_flow_run
flow = await load_flow_from_flow_run(flow_run, client=client)
File "/home/malavika/.virtualenvs/lib/python3.8/site-packages/prefect/client.py", line 103, in with_injected_client
return await fn(*args, **kwargs)
File "/home/malavika/.virtualenvs/lib/python3.8/site-packages/prefect/deployments.py", line 54, in load_flow_from_flow_run
await storage_block.get_directory(from_path=deployment.path, local_path=".")
File "/home/malavika/.virtualenvs/lib/python3.8/site-packages/prefect/filesystems.py", line 134, in get_directory
shutil.copytree(from_path, local_path, dirs_exist_ok=True)
File "/usr/lib/python3.8/shutil.py", line 555, in copytree
with os.scandir(src) as itr:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmpbkxm8cdtprefect'
What can be done to link this flow to the deployment properly and get it to run through the Orion UI?Matthias
09/09/2022, 6:55 AMMichael Law
09/09/2022, 8:05 AMBal Raj
09/09/2022, 10:55 AMgertjan
09/09/2022, 11:23 AMget_run_logger
Viet Nguyen
09/09/2022, 11:24 AMSlackbot
09/09/2022, 12:04 PMVadym Dytyniak
09/09/2022, 12:33 PMModuleNotFoundError: No module named 's3fs'
?Vadym Dytyniak
09/09/2022, 12:44 PMDominik Wagner
09/09/2022, 1:10 PMVadym Dytyniak
09/09/2022, 1:13 PMMark Li
09/09/2022, 1:56 PMBenjamin Cerigo
09/09/2022, 2:25 PMKei Fung Dennis Tai
09/09/2022, 2:36 PMRobin Weiß
09/09/2022, 3:20 PMprefect.exceptions.PrefectHTTPStatusError: Server error '502 Bad Gateway' for url '<https://api.prefect.cloud/api/accounts/><ACCOUNT>/workspaces/<WORKSPACE>/logs/'
For more information check: <https://httpstatuses.com/502>
Worker information:
Approximate queue length: 0
Pending log batch length: 1
Pending log batch size: 360
The log worker will attempt to send these logs again in 2.0s
Is it possible that there are too many log messages sent and the Prefect Cloud API can not respond in time?
Cheers!Amol Shirke
09/09/2022, 4:01 PMJavier Smith
09/09/2022, 4:54 PMMansour Zayer
09/09/2022, 5:35 PMNathaniel Russell
09/09/2022, 5:58 PMIlya Galperin
09/09/2022, 6:11 PMIlya Galperin
09/09/2022, 6:11 PMJean Luciano
09/09/2022, 6:45 PMAgent started! Looking for work from queue(s): workque-chosen...
Ilya Galperin
09/09/2022, 6:47 PMJean Luciano
09/09/2022, 7:17 PMIlya Galperin
09/09/2022, 7:33 PMChristopher Boyd
09/12/2022, 5:13 PMIlya Galperin
09/12/2022, 5:26 PMprefect-agent-8b76465df-sq4g7 agent Starting v2.3.2 agent connected to
prefect-agent-8b76465df-sq4g7 agent <https://api.prefect.cloud/api/accounts/ACCOUNTID/work>
prefect-agent-8b76465df-sq4g7 agent spaces/WORKSPACEID...
prefect-agent-8b76465df-sq4g7 agent
prefect-agent-8b76465df-sq4g7 agent ___ ___ ___ ___ ___ ___ _____ _ ___ ___ _ _ _____
prefect-agent-8b76465df-sq4g7 agent | _ \ _ \ __| __| __/ __|_ _| /_\ / __| __| \| |_ _|
prefect-agent-8b76465df-sq4g7 agent | _/ / _|| _|| _| (__ | | / _ \ (_ | _|| .` | | |
prefect-agent-8b76465df-sq4g7 agent |_| |_|_\___|_| |___\___| |_| /_/ \_\___|___|_|\_| |_|
prefect-agent-8b76465df-sq4g7 agent
prefect-agent-8b76465df-sq4g7 agent
prefect-agent-8b76465df-sq4g7 agent Agent started! Looking for work from queue(s): main...
However, no flows from our “main” queue are being picked up by this agent which makes me think there might be a communication problem between the agent and the server, whether it’s an incorrect API key or something else. Is there some other logs or mechanism we can use to validate that it is connecting successfully… you mentioned API logs?Christopher Boyd
09/12/2022, 6:07 PMIlya Galperin
09/12/2022, 6:09 PMChristopher Boyd
09/12/2022, 6:09 PMIlya Galperin
09/12/2022, 6:10 PMChristopher Boyd
09/12/2022, 6:10 PMIlya Galperin
09/12/2022, 6:11 PMapiVersion: apps/v1
kind: Deployment
metadata:
name: prefect-agent
namespace: prefect2
spec:
selector:
matchLabels:
app: prefect-agent
replicas: 1
template:
metadata:
labels:
app: prefect-agent
spec:
serviceAccountName: default
containers:
- name: agent
image: prefecthq/prefect:${var.prefect_tag}
command: ["prefect", "agent", "start", "-q", "main"]
imagePullPolicy: "IfNotPresent"
env:
- name: PREFECT_API_URL
value: ${var.prefect_api_url}
- name: PREFECT_API_KEY
valueFrom:
secretKeyRef:
name: prefect-cloud
key: api_key
${var.prefect_tag}
in this case resolves to “2.3.2-python3.10”Christopher Boyd
09/12/2022, 6:13 PMIlya Galperin
09/12/2022, 6:13 PMChristopher Boyd
09/12/2022, 6:24 PMapiVersion: apps/v1
kind: Deployment
metadata:
name: orion
namespace: prefect2
spec:
selector:
matchLabels:
app: orion
replicas: 1 # We're using SQLite, so we should only run 1 pod
template:
metadata:
labels:
app: orion
spec:
containers:
- name: api
image: prefecthq/prefect:2.3.2-python3.10
command: ["prefect", "orion", "start", "--host", "0.0.0.0", "--log-level", "WARNING"]
imagePullPolicy: "IfNotPresent"
ports:
- containerPort: 4200
- name: agent
image: prefecthq/prefect:2.3.2-python3.10
command: ["prefect", "agent", "start", "kubernetes"]
imagePullPolicy: "IfNotPresent"
env:
- name: PREFECT_API_URL
value: <https://api.prefect.cloud/api/accounts/><accountid>/workspaces/<workspaceid>
- name: PREFECT_API_KEY
value: bad_value
---
apiVersion: v1
kind: Service
metadata:
name: orion
namespace: prefect2
labels:
app: orion
spec:
ports:
- port: 4200
protocol: TCP
selector:
app: orion
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: Role
metadata:
namespace: prefect2
name: flow-runner
rules:
- apiGroups: [""]
resources: ["pods", "pods/log", "pods/status"]
verbs: ["get", "watch", "list"]
- apiGroups: ["batch"]
resources: ["jobs"]
verbs: [ "get", "list", "watch", "create", "update", "patch", "delete" ]
---
apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: RoleBinding
metadata:
name: flow-runner-role-binding
namespace: prefect2
subjects:
- kind: ServiceAccount
name: default
namespace: prefect2
roleRef:
kind: Role
name: flow-runner
apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
prefect.exceptions.PrefectHTTPStatusError: Client error '401 Unauthorized' for url '<https://api.prefect.cloud/api/accounts/><redact>/workspaces/<redact>/work_queues/name/kubernetes'
Response: {'detail': 'Invalid authentication credentials'}
For more information check: <https://httpstatuses.com/401>
An exception occurred.
Configure Prefect to communicate with the server with:
prefect config set PREFECT_API_URL=<http://0.0.0.0:4200/api>
View the API reference documentation at <http://0.0.0.0:4200/docs>
Check out the dashboard at <http://0.0.0.0:4200>
INFO: Started server process [9]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on <http://0.0.0.0:4200> (Press CTRL+C to quit)
Agents now support multiple work queues. Instead of passing a single argument,
provide work queue names with the `-q` or `--work-queue` flag: `prefect agent
start -q kubernetes`
Starting v2.3.2 agent connected to
<https://api.prefect.cloud/api/accounts/redact/work>
spaces/redact...
___ ___ ___ ___ ___ ___ _____ _ ___ ___ _ _ _____
| _ \ _ \ __| __| __/ __|_ _| /_\ / __| __| \| |_ _|
| _/ / _|| _|| _| (__ | | / _ \ (_ | _|| .` | | |
|_| |_|_\___|_| |___\___| |_| /_/ \_\___|___|_|\_| |_|
Agent started! Looking for work from queue(s): kubernetes...
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 88, in get_work_queues
work_queue = await self.client.create_work_queue(name=name)
File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 835, in create_work_queue
response = await <http://self._client.post|self._client.post>("/work_queues/", json=data)
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1842, in post
return await self.request(
File "/usr/local/lib/python3.10/site-packages/httpx/_client.py", line 1527, in request
return await self.send(request, auth=auth, follow_redirects=follow_redirects)
File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 279, in send
response.raise_for_status()
File "/usr/local/lib/python3.10/site-packages/prefect/client.py", line 225, in raise_for_status
raise PrefectHTTPStatusError.from_httpx_error(exc) from exc.__cause__
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url '<http://google.com/work_queues/>'
For more information check: <https://httpstatuses.com/404>
Ilya Galperin
09/12/2022, 6:33 PMChristopher Boyd
09/12/2022, 6:34 PMIlya Galperin
09/12/2022, 6:34 PMService
manifest here have any impact on connectivity between Cloud and the Agent? My understanding is that this would only be for communication between pods spawned by the agent and the agent itself, right?Starting v2.3.0 agent connected to <http://google.com>...
___ ___ ___ ___ ___ ___ _____ _ ___ ___ _ _ _____
| _ \ _ \ __| __| __/ __|_ _| /_\ / __| __| \| |_ _|
| _/ / _|| _|| _| (__ | | / _ \ (_ | _|| .` | | |
|_| |_|_\___|_| |___\___| |_| /_/ \_\___|___|_|\_| |_|
Agent started! Looking for work from queue(s): main...
Christopher Boyd
09/12/2022, 6:39 PMIlya Galperin
09/12/2022, 6:39 PMChristopher Boyd
09/12/2022, 6:40 PMAgents now support multiple work queues. Instead of passing a single argument,
provide work queue names with the `-q` or `--work-queue` flag: `prefect agent
start -q kubernetes`
Starting v2.3.2 agent connected to <https://google.com>...
___ ___ ___ ___ ___ ___ _____ _ ___ ___ _ _ _____
| _ \ _ \ __| __| __/ __|_ _| /_\ / __| __| \| |_ _|
| _/ / _|| _|| _| (__ | | / _ \ (_ | _|| .` | | |
|_| |_|_\___|_| |___\___| |_| /_/ \_\___|___|_|\_| |_|
Agent started! Looking for work from queue(s): kubernetes...
prefect.exceptions.PrefectHTTPStatusError: Client error '404 Not Found' for url '<https://google.com/work_queues/>'
For more information check: <https://httpstatuses.com/404>
Ilya Galperin
09/12/2022, 6:43 PMChristopher Boyd
09/12/2022, 6:44 PMIlya Galperin
09/12/2022, 6:45 PMChristopher Boyd
09/12/2022, 6:45 PMIlya Galperin
09/12/2022, 6:45 PMChristopher Boyd
09/12/2022, 6:51 PMIlya Galperin
09/12/2022, 6:54 PMChristopher Boyd
09/12/2022, 6:55 PMprefect cloud login --key <key>
Ilya Galperin
09/12/2022, 7:18 PM# apt-get install curl
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package curl
# apt-get update
Err:1 <http://deb.debian.org/debian> bullseye InRelease
Temporary failure resolving '<http://deb.debian.org|deb.debian.org>'
Christopher Boyd
09/12/2022, 7:22 PMcat DOCKERFILE
FROM prefecthq/prefect:2.32.0-python3.10
RUN pip install httpie
then:
export IMAGE_REGISTRY="my_image_registry"
export PROJECT_NAME="prefect-httpie-3.10"
export PROJECT_VERSION="latest"
echo "$IMAGE_REGISTRY/$PROJECT_NAME:$PROJECT_VERSION"
docker build --platform=linux/amd64 -t "$IMAGE_REGISTRY/$PROJECT_NAME:$PROJECT_VERSION" -f ./DOCKERFILE .
docker push "$IMAGE_REGISTRY/$PROJECT_NAME:$PROJECT_VERSION"
Ilya Galperin
09/12/2022, 7:23 PMChristopher Boyd
09/12/2022, 7:24 PMhttp <https://api.prefect.io>
HTTP/1.1 400 Bad Request
Access-Control-Allow-Origin: *
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Length: 18
Content-Type: text/html; charset=utf-8
Date: Mon, 12 Sep 2022 19:24:13 GMT
ETag: W/"12-7JEJwpG8g89ii7CR/6hhfN27Q+k"
Set-Cookie: route=1663010654.252.42.320963|0e12785838f6d88c8042f8a55371e727; Expires=Wed, 14-Sep-22 19:24:13 GMT; Max-Age=172800; Path=/; Secure; HttpOnly
Strict-Transport-Security: max-age=15724800; includeSubDomains
Via: 1.1 google
X-Powered-By: Express
GET query missing.
Ilya Galperin
09/12/2022, 7:25 PMChristopher Boyd
09/12/2022, 7:25 PMIlya Galperin
09/12/2022, 7:25 PM