Mike Cole
08/17/2023, 11:44 PMRunning deployment push steps...
> Running push_to_gcs step...
Unable to load step function: prefect_gcp.deployment.steps.push_to_gcs. Attempting install of prefect-gcp>=0.4.3.
ModuleNotFoundError: No module named 'prefect_gcp.deployment'
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect_gcp.deployment.steps.push_to_gcs
An exception occurred.
More in thread.prefect 2.11.3
prefect-dbt 0.3.1
prefect-gcp 0.4.5
prefect-shell 0.1.5
prefect-sqlalchemy 0.2.4
prefect deploy
and prefect project init --recipe gcs
to set up a deployment, both approaches give me the errors related to prefect_gcp and deployment.# Welcome to your prefect.yaml file! You can you this file for storing and managing
# configuration for deploying your flows. We recommend committing this file to source
# control along with your flow code.
# Generic metadata about this project
name: test
prefect-version: 2.11.3
# build section allows you to manage and build docker images
build: null
# push section allows you to manage if and how this project is uploaded to remote locations
push:
- prefect_gcp.deployment.steps.push_to_gcs:
id: push_code
requires: prefect-gcp>=0.4.3
bucket: data_lake_mpls-311
folder: test
credentials: "{{ prefect.blocks.gcp-credentials.gcp-mpls311 }}"
# pull section allows you to provide instructions for cloning this project in remote locations
pull:
- prefect_gcp.deployment.steps.pull_from_gcs:
id: pull_code
requires: prefect-gcp>=0.4.3
bucket: '{{ push_code.bucket }}'
folder: '{{ pull_code.folder }}'
credentials: "{{ prefect.blocks.gcp-credentials.gcp-mpls311 }}"
# the deployments section allows you to provide configuration for deploying flows
deployments:
- name: gcptest
version: 1
tags: []
description: null
schedule: {}
flow_name: null
entrypoint: test.py:get_repo_info
parameters: {}
work_pool:
name: my-process-pool
work_queue_name: null
job_variables: {}
prefect deploy --name gcptest
is below, I removed a whole bunch of "Requirement already satisified messages"
Running deployment push steps...
> Running push_to_gcs step...
Unable to load step function: prefect_gcp.deployment.steps.push_to_gcs. Attempting install of prefect-gcp>=0.4.3.
Requirement already satisfied: prefect-gcp>=0.4.3 in /home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages (0.4.5)
Traceback (most recent call last):
File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/deployments/steps/core.py", line 124, in run_steps
step_output = await run_step(step, upstream_outputs)
File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/deployments/steps/core.py", line 94, in run_step
step_func = _get_function_for_step(fqn, requires=keywords.get("requires"))
File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/deployments/steps/core.py", line 61, in _get_function_for_step
step_func = import_object(fully_qualified_name)
File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/utilities/importtools.py", line 212, in import_object
module = load_module(module_name)
File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/utilities/importtools.py", line 183, in load_module
return importlib.import_module(module_name)
File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/importlib/_init_.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
File "<frozen importlib._bootstrap>", line 984, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'prefect_gcp.deployment'
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/cli/_utilities.py", line 41, in wrapper
return fn(*args, **kwargs)
File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/utilities/asyncutils.py", line 255, in coroutine_wrapper
return call()
File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 383, in _call_
return self.result()
File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 283, in result
return self.future.result(timeout=timeout)
File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 169, in result
return self.__get_result()
File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
raise self._exception
File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/_internal/concurrency/calls.py", line 346, in _run_async
result = await coro
File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/cli/deploy.py", line 257, in deploy
await _run_single_deploy(
File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/client/utilities.py", line 51, in with_injected_client
return await fn(*args, **kwargs)
File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/cli/deploy.py", line 517, in _run_single_deploy
await run_steps(push_steps, step_outputs, print_function=app.console.print)
File "/home/udengine/anaconda3/envs/mpls311/lib/python3.9/site-packages/prefect/deployments/steps/core.py", line 152, in run_steps
raise StepExecutionError(f"Encountered error while running {fqn}") from exc
prefect.deployments.steps.core.StepExecutionError: Encountered error while running prefect_gcp.deployment.steps.push_to_gcs
An exception occurred.
Jake Kaplan
08/18/2023, 1:58 PMprefect.deployments.xxxx
Mike Cole
08/18/2023, 2:23 PMprefect init
steps to see how that one worked out. Seemed to go better and I noticed the aws deploy seemed to reference plural deployments (prefect_aws.deployments.steps.push_to_s3), but the gcp one referenced a singular deployment (prefect_gcp.deployment.steps.push_to_gcs). I was staring at the prefect-aws and prefect-gcp repositories trying to figure out why prefect-gcp was singular.Jake Kaplan
08/18/2023, 2:24 PMMike Cole
08/18/2023, 2:24 PMprefect project init --recipe gcs
, it's generating the prefect.yaml file with the singular reference, but the aws one generates with the plural reference.Jake Kaplan
08/18/2023, 2:35 PMMike Cole
08/18/2023, 2:36 PM