https://prefect.io logo
Powered by
Title
k

Kevin Grismore

08/26/2022, 7:56 PM
Anyone know why I get 
KeyError: "No class found for dispatch key 'gcs' in registry for type 'Block'."
when trying to run a flow stored on GCS using a GCS storage block?
āœ… 1
the block loads just fine if I execute this code in a python interpreter: 
from prefect.filesystems import GCS

gcs_block = GCS.load("prefect-access")
r

Ryan Peden

08/26/2022, 8:11 PM
Hi, Kevin. Are you using Prefect Cloud, or running the Orion API locally?
k

Kevin Grismore

08/26/2022, 8:11 PM
Prefect Cloud
running a simple logging test flow ā€¢ storage: GCS block ā€¢ infra: Kubernetes Job block
r

Ryan Peden

08/26/2022, 8:12 PM
Thanks for the details. Which Prefect image you using to run the k8s jobs?
k

Kevin Grismore

08/26/2022, 8:14 PM
prefecthq/prefect:2.2.0-python3.9
. It worked okay when it was just this, but using it as a base image with lots of other packages installed, I get this error. So it was clearly something I added to the image, but it's tough to figure out what.
Could be an outdated gcsfs...
I'll try bumping gcsfs to latest and report back
šŸ‘ 1
r

Ryan Peden

08/26/2022, 8:17 PM
Sounds good - thanks
k

Kevin Grismore

08/26/2022, 8:26 PM
That did not fix it. šŸ˜ž
The image I'm using is such a wild mashup of new and old packages, it may be difficult to track down the root cause here.
I know the flow also runs fine locally, so it's strictly an issue with flow retrieval from remote storage.
r

Ryan Peden

08/26/2022, 8:48 PM
Would you be able to share a list of which packages you added to the base image?
k

Kevin Grismore

08/26/2022, 8:49 PM
yep
r

Ryan Peden

08/26/2022, 8:52 PM
Thanks. I know where this is happening in the code, but not why it's happening for a class in prefect.filesystems - so that package list might make this easier to reproduce and figure out
k

Kevin Grismore

08/26/2022, 8:53 PM
it's gonna be a long list šŸ˜¢
here's a thought before I go that far
the requirements.txt I'm installing from also has prefect==2.2.0 in it. Is it a terrible idea to tell pip to install prefect to the prefect base image? Or will it resolve as already installed?
and here is the same env that should be running in the image, note that some of them are not public packages: 
aiofiles==0.8.0
aiohttp==3.8.1
aiosignal==1.2.0
aiosqlite==0.17.0
alembic==1.8.1
anyio==3.6.1
archiver==1.0.11
asgi-lifespan==1.0.1
async-timeout==4.0.2
asyncpg==0.26.0
attrs==22.1.0
awesome-slugify==1.6.5
beautifulsoup4==4.8.2
boto3==1.9.124
botocore==1.12.253
cachetools==4.2.4
camelot-py==0.10.1
cffi==1.15.1
chardet==3.0.4
charset-normalizer==2.1.0
click==8.1.3
cloudpickle==2.1.0
commonmark==0.9.1
coolname==1.1.0
coverage==6.4.3
croniter==1.3.5
cryptography==37.0.4
decorator==5.1.1
distro==1.7.0
docker==6.0.0
docutils==0.15.2
elasticsearch==7.13.1
et-xmlfile==1.1.0
Faker==13.15.1
fastapi==0.79.1
frozenlist==1.3.1
fsspec==2022.7.1
future==0.18.2
fuzzywuzzy==0.17.0
gcsfs==2021.4.0
ghostscript==0.7
google-api-core==2.8.2
google-auth==1.35.0
google-auth-oauthlib==0.5.2
google-cloud-core==1.7.3
google-cloud-storage==1.38.0
google-crc32c==1.3.0
google-resumable-media==1.3.3
googleapis-common-protos==1.56.4
greenlet==1.1.2
griffe==0.21.0
h11==0.12.0
hare==1.2.0
headless==72.0.3618.0+1.gfe69cf8
httpcore==0.15.0
httpx==0.23.0
idna==2.8
importlib-metadata==4.12.0
iniconfig==1.1.1
jmespath==0.10.0
jsonpatch==1.32
jsonpointer==2.3
keyring==23.8.2
keyrings.google-artifactregistry-auth==1.0.0
kubernetes==24.2.0
lxml==4.4.2
lzstring==1.0.4
Mako==1.2.1
MarkupSafe==2.1.1
multidict==6.0.2
nameparser==1.1.1
numpy==1.19.5
oauthlib==3.2.0
olefile==0.46
opencv-python==4.4.0.46
openpyxl==3.0.10
orderedmultidict==1.0.1
orjson==3.5.1
packaging==21.3
pandas==1.4.3
pathspec==0.9.0
pdfminer.six==20201018
pdftopng==0.2.3
pendulum==2.1.2
phonenumbers==8.12.52
pika==1.2.0
Pillow==4.0.0
pluggy==1.0.0
prefect==2.2.0
protobuf==4.21.5
py==1.11.0
pyarrow==9.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyclops==2.1.0
pycparser==2.21
pydantic==1.9.2
Pygments==2.13.0
pyocr==0.7.2
pyOpenSSL==22.0.0
pyparsing==3.0.9
PyPDF2==2.9.0
PySocks==1.7.1
pyssdb==0.4.2
pytest==7.1.2
pytest-cov==3.0.0
python-dateutil==2.8.2
python-dotenv==0.17.1
python-Levenshtein==0.12.0
python-slugify==6.1.2
pytz==2022.2.1
pytzdata==2020.1
pyxlsb==1.0.9
PyYAML==6.0
readchar==4.0.2
regex==2022.7.25
requests==2.28.1
requests-ftp==0.3.1
requests-oauthlib==1.3.1
requests-toolbelt==0.9.1
rfc3986==1.5.0
rich==12.5.1
rosetta==0.2.2
rsa==4.9
Rtree==0.9.3
s3transfer==0.2.1
selenium==3.3.1
six==1.16.0
slack-sdk==3.18.1
sniffio==1.2.0
sortedcontainers==2.4.0
soupsieve==2.3.2.post1
SQLAlchemy==1.4.40
starlette==0.19.1
stem==1.8.0
tabula-py==2.4.0
tabulate==0.8.10
tabulawrapper==2.0+5.gbf09143
text-unidecode==1.3
tika==1.23
toml==0.10.2
tomli==2.0.1
typer==0.6.1
typing_extensions==4.3.0
Unidecode==0.4.21
urllib3==1.26.12
uvicorn==0.18.2
websocket-client==1.3.3
xlrd==1.1.0
yarl==1.8.1
ziplus==1.0+8.ga022d63
zipp==3.8.1
I did also notice that the 
GCS
class in 
prefect.filesystems
is the only one that doesn't have 
_block_type_name
defined. Not sure if that has anything to do with this though.
r

Ryan Peden

08/26/2022, 9:03 PM
Blocks fall back to using the class name if 
_block_type_name
isn't specified so I don't think that is it
k

Kevin Grismore

08/26/2022, 9:04 PM
That's what I assumed as soon as I posted that lol
it's gotta be a dependency conflict pip didn't catch given the flow works fine on the clean prefect image
doing a pip freeze on a brand new env with just prefect==2.2.0 installed and comparing against a pip freeze on my env, and noting the differences for prefect's reqs. will update again
my env left, fresh prefect install right 
cachetools==4.2.4 vs cachetools==5.2.0
charset-normalizer==2.1.0 vs charset-normalizer==2.1.1
fastapi==0.79.1 vs fastapi==0.81.0
greenlet==1.1.2 vs greenlet==1.1.3
idna==2.8 vs idna==3.3
websocket-client==1.3.3 vs websocket-client==1.4.0
though it appears some of these are a matter of when I made my existing local env
r

Ryan Peden

08/26/2022, 9:47 PM
thanks, Kevin - I'm reaching out to some of my colleagues here to see if they know the answer to this
ā¤ļø 1
a

Anna Geller

08/27/2022, 1:09 AM
Kevin, can you get into a bash of the running container and investigate there whether you can import prefect and gcsfs fine and whether the versions are OK? Also, could you share how you created the deployment with this block? If nothing else works it would be helpful if you could create a GitHub issue with an MRE so that we can try and reproduce, and troubleshoot further if needed
k

Kevin Grismore

08/29/2022, 3:26 PM
hi anna, thanks for the advice here. I'm running the image in a local container to check these things. Prefect and gcsfs are both on their latest versions and import successfully in python. I can even write the same prefect flow in the container and run it successfully
I'll keep trying things and see what I can come up with
ahhhh I think I found it. prefect uses 
python-slugify
my project uses 
awesome-slugify
both import as 
slugify
I discovered this by trying to start an agent using my image, and the request failed because the slug in the body wasn't formatted correctly.
r

Ryan Peden

08/29/2022, 3:40 PM
That sounds right, since the default behavior of blocks is to use a slugify-generated slug to register the class
k

Kevin Grismore

08/29/2022, 3:42 PM
yep. well that's fun.
I haven't run into a same-name package conflict like this before, so I'll do some thinking about how I can resolve it. Thanks for digging in with me, I really appreciate it
r

Ryan Peden

08/29/2022, 4:05 PM
You're welcome! Thanks for the work you put in to track this down.
3 Views
#prefect-communityJoin Slack
Powered by