https://prefect.io logo
Channels
Powered by
m

Mike Cole

08/17/2023, 11:44 PM
Anyone familiar with prefect-gcp and doing deployments, at least with the latest versions of prefect (2.11.3) and prefect-gcp (0.4.5)? My end goal was to have my flow code stored in a GCS bucket. I was having trouble, so I started from a blank slate with a new VM, new prefect install, etc. When I attempt to deploy a flow, I'm getting errors like:
Copy code
Running deployment push steps...
 > Running push_to_gcs step...
Unable to load step function: prefect_gcp.deployment.steps.push_to_gcs. Attempting install of prefect-gcp>=0.4.3.
Copy code
ModuleNotFoundError: No module named 'prefect_gcp.deployment'
Copy code
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect_gcp.deployment.steps.push_to_gcs
An exception occurred.
More in thread.
✅ 1
I have a VM with prefect running on it, self-hosted based on the instructions at this link, using postgres for the DB https://docs.prefect.io/2.11.3/guides/host/ PIP version snapshot below for prefect:
Copy code
prefect                       2.11.3
prefect-dbt                   0.3.1
prefect-gcp                   0.4.5
prefect-shell                 0.1.5
prefect-sqlalchemy            0.2.4
I took the simple task/flow setup from the tutorial (https://docs.prefect.io/2.11.3/tutorial/tasks/) and was able to deploy that fine. I have GCP and GCS blocks created. I'm tried using both 
prefect deploy
and 
prefect project init --recipe gcs
to set up a deployment, both approaches give me the errors related to prefect_gcp and deployment.
My prefect.yaml file from the prefect init looks like:
Copy code
# Welcome to your prefect.yaml file! You can you this file for storing and managing
# configuration for deploying your flows. We recommend committing this file to source
# control along with your flow code.

# Generic metadata about this project
name: test
prefect-version: 2.11.3

# build section allows you to manage and build docker images
build: null

# push section allows you to manage if and how this project is uploaded to remote locations
push:
- prefect_gcp.deployment.steps.push_to_gcs:
    id: push_code
    requires: prefect-gcp>=0.4.3
    bucket: data_lake_mpls-311
    folder: test
    credentials: "{{ prefect.blocks.gcp-credentials.gcp-mpls311 }}"

# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect_gcp.deployment.steps.pull_from_gcs:
    id: pull_code
    requires: prefect-gcp>=0.4.3
    bucket: '{{ push_code.bucket }}'
    folder: '{{ pull_code.folder }}'
    credentials: "{{ prefect.blocks.gcp-credentials.gcp-mpls311 }}"

# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: gcptest
  version: 1
  tags: []
  description: null
  schedule: {}
  flow_name: null
  entrypoint: test.py:get_repo_info
  parameters: {}
  work_pool:
    name: my-process-pool
    work_queue_name: null
    job_variables: {}
The full message I get when I run 
prefect deploy --name gcptest
is below, I removed a whole bunch of "Requirement already satisified messages"
Copy code
Running deployment push steps...
 > Running push_to_gcs step...
Unable to load step function: prefect_gcp.deployment.steps.push_to_gcs. Attempting install of prefect-gcp>=0.4.3.
Requirement already satisfied: prefect-gcp>=0.4.3 in /home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages (0.4.5)

Traceback (most recent call last):
  File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/deployments/steps/core.py", line 124, in run_steps
    step_output = await run_step(step, upstream_outputs)
  File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/deployments/steps/core.py", line 94, in run_step
    step_func = _get_function_for_step(fqn, requires=keywords.get("requires"))
  File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/deployments/steps/core.py", line 61, in _get_function_for_step
    step_func = import_object(fully_qualified_name)
  File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/utilities/importtools.py", line 212, in import_object
    module = load_module(module_name)
  File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/utilities/importtools.py", line 183, in load_module
    return importlib.import_module(module_name)
  File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/importlib/_init_.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'prefect_gcp.deployment'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
    return fn(*args, **kwargs)
  File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 255, in coroutine_wrapper
    return call()
  File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 383, in _call_
    return self.result()
  File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 283, in result
    return self.future.result(timeout=timeout)
  File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 169, in result
    return self.__get_result()
  File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 346, in _run_async
    result = await coro
  File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/cli/deploy.py", line 257, in deploy
    await _run_single_deploy(
  File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
    return await fn(*args, **kwargs)
  File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/cli/deploy.py", line 517, in _run_single_deploy
    await run_steps(push_steps, step_outputs, print_function=app.console.print)
  File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/deployments/steps/core.py", line 152, in run_steps
    raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect_gcp.deployment.steps.push_to_gcs
An exception occurred.
I'm stumped, can't figure out how to do deployments to push/pull flow code to a GCS bucket. Hoping someone might have an idea of how to troubleshoot this further.
j

Jake Kaplan

08/18/2023, 1:58 PM
Hey, I believe you need to make your step pluralized:
Copy code
prefect.deployments.xxxx
upvote 1
m

Mike Cole

08/18/2023, 2:23 PM
Thanks so much Jake, that helped me figure it out, and it looks like I can successfully build the deployment now. I had also pip installed prefect-aws, and gone through some similar 
prefect init
steps to see how that one worked out. Seemed to go better and I noticed the aws deploy seemed to reference plural deployments (prefect_aws.deployments.steps.push_to_s3), but the gcp one referenced a singular deployment (prefect_gcp.deployment.steps.push_to_gcs). I was staring at the prefect-aws and prefect-gcp repositories trying to figure out why prefect-gcp was singular.
j

Jake Kaplan

08/18/2023, 2:24 PM
ah I see the recipe has the singular version, I'll get that updated
sorry for the confusion, glad it's working!
m

Mike Cole

08/18/2023, 2:24 PM
Didn't even think to look at the prefect.yaml file that gets generated with prefect init. Looks like with 
prefect project init --recipe gcs
, it's generating the prefect.yaml file with the singular reference, but the aws one generates with the plural reference.
Yep, that looks to be the issue. Thanks!
FYI, I did take a peek at the prefect-gcp docs out at https://prefecthq.github.io/prefect-gcp/deployments/steps/, and everything looks correct there, referencing prefect_gcp.deployments.
j

Jake Kaplan

08/18/2023, 2:35 PM
fixed here: https://github.com/PrefectHQ/prefect/pull/10442 will be in the next release
m

Mike Cole

08/18/2023, 2:36 PM
Very cool, thanks!
8 Views
Powered by