https://prefect.io logo
Title
t

Thomas Fredriksen

09/30/2022, 5:59 PM
Hi there! I was struggling to deploy a simple flow from code using
KubernetesJob
. However, just before posting a question in this channel, I figured I should "give it one last try" and to my surprise it actually worked 🤡 As the post is already written, I figured I might as well share was I experienced and the solutions so that anyone else having simiular issues may have somewhere it start. I was trying to create a deployment using
KubernetesJob
as infrastructure. The deployment looked like this (see code further down in the thread) The container image is built from a dockerfile based on
prefecthq/prefect:2-latest
and copies the script containing the flow to
/opt/prefect/flows/flow.py
. This deploys just fine, and I can see that the kubernetes job is created and the Prefect engine starts, however it does not seem like the flow runst, and I instead get the following error (see traceback further down in the thread) It turns out that the base docker-image I had chosen was the issue, as the prefect version did not match the version the server was running (
2.4.5
vs
2.4.2
). From the release-notes for
2.4.3
-
2.4.5
i see that block protection is an issue, and I speculate that this might be the root cause. Hope it helps 🙂 Edit: Moved code and traceback to the thread
:thank-you: 1
✅ 1
z

Zanie

09/30/2022, 6:03 PM
Hi! Thanks for the detailed walkthrough here 🙂 If your server is running on 2.4.5 you shouldn’t have encountered this issue even if the client was on an old version.
For reference, we’ve got integration tests covering this in https://github.com/PrefectHQ/prefect/actions/runs/3154346252/jobs/5131837899
🙌 1
j

Jeff Hale

09/30/2022, 6:09 PM
Thanks, Thomas. Glad to hear it’s working. 🙂 Could you please add the code and traceback to this thread to keep the main channel tidy?
t

Thomas Fredriksen

09/30/2022, 6:12 PM
Deployment code:
deployment = Deployment.build_from_flow(
    flow=flow,
    skip_upload=dry_run,
    entrypoint="/opt/prefect/flows/{}:{}".format(os.path.basename(context["flow_file"]), "main"),
    name=metadata["name"],
    work_queue_name=metadata.get("workQueue", "default"),
    infrastructure=KubernetesJob(
        image="<http://my-registry.com/myimage:latest|my-registry.com/myimage:latest>",
        env={
            "PREFECT_LOGGING_LEVEL": "DEBUG"
        },
        labels={
            "<http://prefect.io/flowName|prefect.io/flowName>": self._flow.name,
            "<http://prefect.io/flowVersion|prefect.io/flowVersion>": self._flow.version,
            "<http://prefect.io/taskRunner|prefect.io/taskRunner>": self._flow.task_runner.name,
        },
        name=f"{self._flow.name}-{self._flow.version}",
        namespace="prefect",
        finished_job_ttl=10,
        customizations=JsonPatch([
            {
                "op": "add",
                "path": "/spec/template/spec/containers/0/resources",
                "value": {
                    "requests": {
                        "cpu": tier.cpu,
                        "memory": tier.mem
                    },
                    "limits": {
                        "cpu": tier.cpu,
                        "memory": tier.mem
                    }
                },
            }
        ]),
        stream_output=True,
    ),
)

deployment.apply()
Traceback error:
12:35:54.284 | DEBUG   | prefect.client - Connecting to API at <http://orion:4200/api/>
12:35:54.311 | DEBUG   | Flow run 'jolly-crane' - Loading flow for deployment 'hello-cloud'...
12:35:54.328 | DEBUG   | Flow run 'jolly-crane' - Starting 'ConcurrentTaskRunner'; submitted tasks will be run concurrently...
12:35:54.329 | DEBUG   | prefect.task_runner.concurrent - Starting task runner...
12:35:54.338 | DEBUG   | prefect.client - Connecting to API at <http://orion:4200/api/>
12:35:54.429 | DEBUG   | prefect.task_runner.concurrent - Shutting down task runner...
12:35:54.429 | ERROR   | Flow run 'jolly-crane' - Crash detected! Execution was interrupted by an unexpected exception.
12:35:54.430 | DEBUG   | Flow run 'jolly-crane' - Crash details:
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1350, in report_flow_run_crashes
    yield
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 365, in begin_flow_run
    await result_filesystem._save(is_anonymous=True)
  File "/usr/local/lib/python3.10/site-packages/prefect/client/orion.py", line 82, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 769, in _save
    block_document=self._to_block_document(name=name)
  File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 339, in _to_block_document
    raise ValueError(
ValueError: No block schema ID provided, either as an argument or on the block.
12:35:54.451 | DEBUG   | prefect.engine - Reported crashed flow run 'jolly-crane' successfully!
12:35:54.475 | ERROR   | prefect.engine - Engine execution of flow run '05dff8be-c6bb-4d3f-95f8-491074fdf99e' exited with unexpected exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 1620, in <module>
    enter_flow_run_engine_from_subprocess(flow_run_id)
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 183, in enter_flow_run_engine_from_subprocess
    return anyio.run(retrieve_flow_then_begin_flow_run, flow_run_id)
  File "/usr/local/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 70, in run
    return asynclib.run(func, *args, **backend_options)
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 292, in run
    return native_run(wrapper(), debug=debug)
  File "/usr/local/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.10/asyncio/base_events.py", line 646, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 287, in wrapper
    return await func(*args)
  File "/usr/local/lib/python3.10/site-packages/prefect/client/orion.py", line 82, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 306, in retrieve_flow_then_begin_flow_run
    return await begin_flow_run(
  File "/usr/local/lib/python3.10/site-packages/prefect/engine.py", line 365, in begin_flow_run
    await result_filesystem._save(is_anonymous=True)
  File "/usr/local/lib/python3.10/site-packages/prefect/client/orion.py", line 82, in with_injected_client
    return await fn(*args, **kwargs)
  File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 769, in _save
    block_document=self._to_block_document(name=name)
  File "/usr/local/lib/python3.10/site-packages/prefect/blocks/core.py", line 339, in _to_block_document
    raise ValueError(
ValueError: No block schema ID provided, either as an argument or on the block.
12:35:54.488 | DEBUG   | prefect.client - Connecting to API at <http://orion:4200/api/>
:thank-you: 1