Thomas Fredriksen
09/30/2022, 5:59 PMKubernetesJob
. However, just before posting a question in this channel, I figured I should "give it one last try" and to my surprise it actually worked 🤡
As the post is already written, I figured I might as well share was I experienced and the solutions so that anyone else having simiular issues may have somewhere it start.
I was trying to create a deployment using KubernetesJob
as infrastructure. The deployment looked like this (see code further down in the thread)
The container image is built from a dockerfile based on prefecthq/prefect:2-latest
and copies the script containing the flow to /opt/prefect/flows/flow.py
.
This deploys just fine, and I can see that the kubernetes job is created and the Prefect engine starts, however it does not seem like the flow runst, and I instead get the following error (see traceback further down in the thread)
It turns out that the base docker-image I had chosen was the issue, as the prefect version did not match the version the server was running (2.4.5
vs 2.4.2
). From the release-notes for 2.4.3
-2.4.5
i see that block protection is an issue, and I speculate that this might be the root cause.
Hope it helps 🙂
Edit: Moved code and traceback to the threadZanie
Jeff Hale
09/30/2022, 6:09 PMThomas Fredriksen
09/30/2022, 6:12 PMdeployment = Deployment.build_from_flow(
flow=flow,
skip_upload=dry_run,
entrypoint="/opt/prefect/flows/{}:{}".format(os.path.basename(context["flow_file"]), "main"),
name=metadata["name"],
work_queue_name=metadata.get("workQueue", "default"),
infrastructure=KubernetesJob(
image="<http://my-registry.com/myimage:latest|my-registry.com/myimage:latest>",
env={
"PREFECT_LOGGING_LEVEL": "DEBUG"
},
labels={
"<http://prefect.io/flowName|prefect.io/flowName>": self._flow.name,
"<http://prefect.io/flowVersion|prefect.io/flowVersion>": self._flow.version,
"<http://prefect.io/taskRunner|prefect.io/taskRunner>": self._flow.task_runner.name,
},
name=f"{self._flow.name}-{self._flow.version}",
namespace="prefect",
finished_job_ttl=10,
customizations=JsonPatch([
{
"op": "add",
"path": "/spec/template/spec/containers/0/resources",
"value": {
"requests": {
"cpu": tier.cpu,
"memory": tier.mem
},
"limits": {
"cpu": tier.cpu,
"memory": tier.mem
}
},
}
]),
stream_output=True,
),
)
deployment.apply()
Traceback error:
12:35:54.284 | DEBUG | prefect.client - Connecting to API at <http://orion:4200/api/>
12:35:54.311 | DEBUG | Flow run 'jolly-crane' - Loading flow for deployment 'hello-cloud'...
12:35:54.328 | DEBUG | Flow run 'jolly-crane' - Starting 'ConcurrentTaskRunner'; submitted tasks will be run concurrently...
12:35:54.329 | DEBUG | prefect.task_runner.concurrent - Starting task runner...
12:35:54.338 | DEBUG | prefect.client - Connecting to API at <http://orion:4200/api/>
12:35:54.429 | DEBUG | prefect.task_runner.concurrent - Shutting down task runner...
12:35:54.429 | ERROR | Flow run 'jolly-crane' - Crash detected! Execution was interrupted by an unexpected exception.
12:35:54.430 | DEBUG | Flow run 'jolly-crane' - Crash details:
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1350, in report_flow_run_crashes
yield
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 365, in begin_flow_run
await result_filesystem._save(is_anonymous=True)
File "/usr/local/lib/python3.10/site-packages/prefect/client/orion.py", line 82, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 769, in _save
block_document=self._to_block_document(name=name)
File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 339, in _to_block_document
raise ValueError(
ValueError: No block schema ID provided, either as an argument or on the block.
12:35:54.451 | DEBUG | prefect.engine - Reported crashed flow run 'jolly-crane' successfully!
12:35:54.475 | ERROR | prefect.engine - Engine execution of flow run '05dff8be-c6bb-4d3f-95f8-491074fdf99e' exited with unexpected exception
Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1620, in <module>
enter_flow_run_engine_from_subprocess(flow_run_id)
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 183, in enter_flow_run_engine_from_subprocess
return anyio.run(retrieve_flow_then_begin_flow_run, flow_run_id)
File "/usr/local/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
return asynclib.run(func, *args, **backend_options)
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
return native_run(wrapper(), debug=debug)
File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
File "/usr/local/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
return future.result()
File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
return await func(*args)
File "/usr/local/lib/python3.10/site-packages/prefect/client/orion.py", line 82, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 306, in retrieve_flow_then_begin_flow_run
return await begin_flow_run(
File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 365, in begin_flow_run
await result_filesystem._save(is_anonymous=True)
File "/usr/local/lib/python3.10/site-packages/prefect/client/orion.py", line 82, in with_injected_client
return await fn(*args, **kwargs)
File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 769, in _save
block_document=self._to_block_document(name=name)
File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 339, in _to_block_document
raise ValueError(
ValueError: No block schema ID provided, either as an argument or on the block.
12:35:54.488 | DEBUG | prefect.client - Connecting to API at <http://orion:4200/api/>