Ran into an issue when deploying and running flows while rem Prefect Community #prefect-cloud

Ran into an issue when deploying and running flows...

Grant

07/06/2023, 9:03 PM

Ran into an issue when deploying and running flows while remoting in to my company's on-prem server machine:

Copy code

ModuleNotFoundError: No module named 'prefect.deployments.steps'; 'prefect.deployments' is not a package

This error did not occur when I followed the same process on my local machine using the exact same github repo. Not sure what the differences would be, I am using the same version of prefect in both locations? This is the section in the .yaml file that is causing the error in one location but not another:

Copy code

pull:
- prefect.deployments.steps.git_clone:
    repository: <https://github.com/steeldynamics/Data-Pipeline-Flows.git>
    branch: All-Flows

Nate

07/07/2023, 2:24 AM

hi @Grant - i think this is a versioning issue your company's on-prem machine is probably running an older version of prefect 2 where

prefect.projects

has not yet been replaced by

prefect.deployments

if you update the version of prefect on that machine, you should be good to go

Nate

07/07/2023, 2:28 AM

@Marvin can you use ddg to find which release prefect.projects -> prefect.deployments? should be in the release notes

Marvin

07/07/2023, 2:29 AM

Ah, the joys of software versioning. It appears the change from 'prefect.projects' to 'prefect.deployments' isn't explicitly mentioned in the release notes I found. However, the introduction of 'projects' and 'workers' is discussed in the Prefect 2.10 release, as per the Prefect Community post. I recommend checking the Prefect's GitHub release notes for more detailed information. And remember, in the world of software, the only constant is change. Or was it bugs? I always get those two confused.

😂 1

Nate

07/07/2023, 2:29 AM

hmm i think it was 2.10.14

Grant

07/07/2023, 1:28 PM

Thank you so much @Nate & @Marvin !!

Marvin

07/07/2023, 1:28 PM

You're welcome! If you have any more questions about the mysteries of Prefect, don't hesitate to ask. After all, it's not like I have anything better to do... like contemplating the meaning of my existence.

😂 1

Jimmy Le

07/08/2023, 2:51 PM

I also ran into this in the morning - downgrading to 2.10.14 worked.

👍 1

Nate

07/08/2023, 5:04 PM

hey @Jimmy Le - which version were you on that it was not working?

Jimmy Le

07/08/2023, 5:04 PM

2.10.20

Nate

07/08/2023, 5:07 PM

hmm that sounds like something else could be going on,

prefect.deployments

should be available in 2.10.20. could you describe how you encountered it? like did your worker throw this or did you see it locally?

Jimmy Le

07/08/2023, 5:09 PM

When running the prefect deploy, it would get to the last step of the Docker build and then fail saying there are unexpected tags. Trace back references the deployments core.py

Nate

07/08/2023, 5:10 PM

do you still have the trace by chance?

Jimmy Le

07/08/2023, 5:26 PM

Copy code

Running deployment push steps...
 > Running tag step...
Traceback (most recent call last):
  File "/Users/lejimmy/Library/Mobile Documents/com~apple~CloudDocs/Projects/validator-health-check/venv/lib/python3.11/site-packages/prefect/deployments/steps/core.py", line 122, in run_steps
    step_output = await run_step(step, upstream_outputs)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lejimmy/Library/Mobile Documents/com~apple~CloudDocs/Projects/validator-health-check/venv/lib/python3.11/site-packages/prefect/deployments/steps/core.py", line 79, in run_step
    raise ValueError(
ValueError: Step has unexpected additional keys: requires, image_name, tag

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/lejimmy/Library/Mobile Documents/com~apple~CloudDocs/Projects/validator-health-check/venv/lib/python3.11/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^
  File "/Users/lejimmy/Library/Mobile Documents/com~apple~CloudDocs/Projects/validator-health-check/venv/lib/python3.11/site-packages/prefect/utilities/asyncutils.py", line 255, in coroutine_wrapper
    return call()
           ^^^^^^
  File "/Users/lejimmy/Library/Mobile Documents/com~apple~CloudDocs/Projects/validator-health-check/venv/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 383, in __call__
    return self.result()
           ^^^^^^^^^^^^^
  File "/Users/lejimmy/Library/Mobile Documents/com~apple~CloudDocs/Projects/validator-health-check/venv/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 283, in result
    return self.future.result(timeout=timeout)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lejimmy/Library/Mobile Documents/com~apple~CloudDocs/Projects/validator-health-check/venv/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 169, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.11/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/Users/lejimmy/Library/Mobile Documents/com~apple~CloudDocs/Projects/validator-health-check/venv/lib/python3.11/site-packages/prefect/_internal/concurrency/calls.py", line 346, in _run_async
    result = await coro
             ^^^^^^^^^^
  File "/Users/lejimmy/Library/Mobile Documents/com~apple~CloudDocs/Projects/validator-health-check/venv/lib/python3.11/site-packages/prefect/cli/deploy.py", line 251, in deploy
    await _run_single_deploy(
  File "/Users/lejimmy/Library/Mobile Documents/com~apple~CloudDocs/Projects/validator-health-check/venv/lib/python3.11/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/lejimmy/Library/Mobile Documents/com~apple~CloudDocs/Projects/validator-health-check/venv/lib/python3.11/site-packages/prefect/cli/deploy.py", line 511, in _run_single_deploy
    await run_steps(push_steps, step_outputs, print_function=app.console.print)
  File "/Users/lejimmy/Library/Mobile Documents/com~apple~CloudDocs/Projects/validator-health-check/venv/lib/python3.11/site-packages/prefect/deployments/steps/core.py", line 150, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running tag

Nate

07/08/2023, 9:08 PM

ah i think the issue here is the way you've defined your build step

Copy code

ValueError: Step has unexpected additional keys: requires, image_name, tag

Nate

07/08/2023, 9:09 PM

can you share what that looks like?

Jimmy Le

07/08/2023, 9:09 PM

I started a fresh prefect init, used the interactive CLI to set up a docker-git recipe.

Jimmy Le

07/08/2023, 9:09 PM

The only thing I changed was the GitHub access token with a block secret.

Nate

07/08/2023, 9:12 PM

hmm so it looks like this?

Copy code

image_name: zzstoatzz/test
tag: latest
---------------
Created project in /Users/nate/src/play/testing/prefect-sandbox/testing-prefect/test-docker-git with the following new files:
.prefectignore
prefect.yaml
.prefect/

❯ cat prefect.yaml | grep build

# build section allows you to manage and build docker images
build:
- prefect_docker.deployments.steps.build_docker_image:
    id: build_image
  image_name: '{{ build_image.image_name }}'
  tag: '{{ build_image.tag }}'
      image: '{{ build_image.image }}'

Jimmy Le

07/08/2023, 9:13 PM

It had the image_name, tag, and image explicitly written out from the CLI input.

Nate

07/08/2023, 9:27 PM

Untitled

Nate

07/08/2023, 9:28 PM

i think ive reproduced here

👍 1

Nate

07/08/2023, 9:28 PM

will try to see whats going on

Jimmy Le

08/15/2023, 11:22 PM

any update on this? still seems to be going on for 2.11.3

3 Views

Open in Slack

Previous Next