Miguel Moncada
01/04/2023, 11:36 AMSubmission failed. KeyError: "No class found for dispatch key 'cloud-run-job' in registry for type 'Block'."
Probably it's due to a misconfiguration on the deployment python file (or maybe the infrastructure block itself?), adding more info to the thread π€from prefect.deployments import Deployment
from prefect_gcp.cloud_run import CloudRunJob
from prefect.filesystems import GCS
from prefect.orion.schemas.schedules import CronSchedule
from dataflows.flows.hello_flow import hello
storage = GCS.load("default")
infrastructure = CloudRunJob("default")
def deploy_hello_flow():
deployment = Deployment.build_from_flow(
flow=hello,
name="hello_flow_deployment",
work_queue_name="default",
storage=storage,
path="hello",
infrastructure=infrastructure,
schedule=CronSchedule(cron="* * * * *", timezone="UTC"),
)
deployment.apply()
if __name__ == "__main__":
deploy_hello_flow()
prefect_gcp
blocks:
β― prefect block register -m prefect_gcp
Successfully registered 6 blocks
βββββββββββββββββββββββββββββββββ
β Registered Blocks β
β‘ββββββββββββββββββββββββββββββββ©
β BigQuery Warehouse β
β GCP Cloud Run Job β
β GCP Credentials β
β GcpSecret β
β GCS Bucket β
β Vertex AI Custom Training Job β
βββββββββββββββββββββββββββββββββ
To configure the newly registered blocks, go to the Blocks page in the Prefect UI.
prefect-gcp
is installed in the docker container where the agent is running:
root@prefect-cloud-vm:/opt/prefect# pip freeze
aiosqlite==0.18.0
alembic==1.9.1
anyio==3.6.2
apprise==1.2.1
asgi-lifespan==2.0.0
asyncpg==0.27.0
cachetools==5.2.0
certifi==2022.12.7
cffi==1.15.1
charset-normalizer==2.1.1
click==8.1.3
cloudpickle==2.2.0
colorama==0.4.6
commonmark==0.9.1
coolname==2.1.0
croniter==1.3.8
cryptography==38.0.4
dask==2022.12.1
dataflows @ file:///opt/prefect
dateparser==1.1.5
distributed==2022.12.1
docker==6.0.1
fastapi==0.88.0
fsspec==2022.11.0
google-api-core==2.11.0
google-api-python-client==2.70.0
google-auth==2.15.0
google-auth-httplib2==0.1.0
googleapis-common-protos==1.57.0
greenlet==2.0.1
griffe==0.25.2
h11==0.14.0
h2==4.1.0
HeapDict==1.0.1
hpack==4.0.0
httpcore==0.16.3
httplib2==0.21.0
httpx==0.23.1
hyperframe==6.0.1
idna==3.4
Jinja2==3.1.2
jsonpatch==1.32
jsonpointer==2.3
kubernetes==25.3.0
locket==1.0.0
Mako==1.2.4
Markdown==3.4.1
MarkupSafe==2.1.1
msgpack==1.0.4
oauthlib==3.2.2
orjson==3.8.3
packaging==22.0
partd==1.3.0
pathspec==0.10.3
pendulum==2.1.2
prefect @ file:///opt/prefect/dist/prefect.tar.gz
prefect-dask==0.2.2
prefect-gcp==0.2.2
protobuf==4.21.12
psutil==5.9.4
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.21
pydantic==1.10.3
Pygments==2.13.0
pyparsing==3.0.9
python-dateutil==2.8.2
python-slugify==7.0.0
pytz==2022.7
pytz-deprecation-shim==0.1.0.post0
pytzdata==2020.1
PyYAML==6.0
readchar==4.0.3
regex==2022.10.31
requests==2.28.1
requests-oauthlib==1.3.1
rfc3986==1.5.0
rich==12.6.0
rsa==4.9
six==1.16.0
sniffio==1.3.0
sortedcontainers==2.4.0
SQLAlchemy==1.4.45
starlette==0.22.0
tblib==1.7.0
text-unidecode==1.3
toml==0.10.2
toolz==0.12.0
tornado==6.2
typer==0.7.0
typing_extensions==4.4.0
tzdata==2022.7
tzlocal==4.2
uritemplate==4.1.1
urllib3==1.26.13
uvicorn==0.20.0
websocket-client==1.4.2
zict==2.2.0
prefect_gcp
from the VM's container was failing (I'm using this Dockerfile & requirements.txt for my agent and the container was apparently missing prefect_gcp
submodules bigquery
, cloud_storage
, cloud_run
)prefect_gcp
and registering the blocks
root@prefect-cloud-vm:/opt/prefect# prefect block register -m prefect_gcp
Successfully registered 6 blocks
βββββββββββββββββββββββββββββββββ
β Registered Blocks β
β‘ββββββββββββββββββββββββββββββββ©
β BigQuery Warehouse β
β GCP Cloud Run Job β
β GCP Credentials β
β GcpSecret β
β GCS Bucket β
β Vertex AI Custom Training Job β
βββββββββββββββββββββββββββββββββ
To configure the newly registered blocks, go to the Blocks page in the Prefect UI.
root@prefect-cloud-vm:/opt/prefect# python
Python 3.10.9 (main, Dec 21 2022, 08:51:48) [GCC 10.2.1 20210110] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import prefect_gcp
>>>
Joseph Desjardins
01/04/2023, 4:36 PMNelson Griffiths
01/18/2023, 10:25 PMMiguel Moncada
01/19/2023, 8:12 AMNelson Griffiths
01/19/2023, 2:25 PMMiguel Moncada
01/19/2023, 2:28 PMNelson Griffiths
01/19/2023, 2:59 PM