https://prefect.io logo
Join Slack
Powered by
Anyone know why I get `KeyError: "No class found f...
# prefect-community
k
Anyone know why I get 
KeyError: "No class found for dispatch key 'gcs' in registry for type 'Block'."
when trying to run a flow stored on GCS using a GCS storage block?
āœ… 1
the block loads just fine if I execute this code in a python interpreter:
Copy code
from prefect.filesystems import GCS

gcs_block = GCS.load("prefect-access")
r
Hi, Kevin. Are you using Prefect Cloud, or running the Orion API locally?
k
Prefect Cloud
running a simple logging test flow ā€¢ storage: GCS block ā€¢ infra: Kubernetes Job block
r
Thanks for the details. Which Prefect image you using to run the k8s jobs?
k
prefecthq/prefect:2.2.0-python3.9
. It worked okay when it was just this, but using it as a base image with lots of other packages installed, I get this error. So it was clearly something I added to the image, but it's tough to figure out what.
Could be an outdated gcsfs...
I'll try bumping gcsfs to latest and report back
šŸ‘ 1
r
Sounds good - thanks
k
That did not fix it. šŸ˜ž
The image I'm using is such a wild mashup of new and old packages, it may be difficult to track down the root cause here.
I know the flow also runs fine locally, so it's strictly an issue with flow retrieval from remote storage.
r
Would you be able to share a list of which packages you added to the base image?
k
yep
r
Thanks. I know where this is happening in the code, but not why it's happening for a class in prefect.filesystems - so that package list might make this easier to reproduce and figure out
k
it's gonna be a long list šŸ˜¢
here's a thought before I go that far
the requirements.txt I'm installing from also has prefect==2.2.0 in it. Is it a terrible idea to tell pip to install prefect to the prefect base image? Or will it resolve as already installed?
and here is the same env that should be running in the image, note that some of them are not public packages:
Copy code
aiofiles==0.8.0
aiohttp==3.8.1
aiosignal==1.2.0
aiosqlite==0.17.0
alembic==1.8.1
anyio==3.6.1
archiver==1.0.11
asgi-lifespan==1.0.1
async-timeout==4.0.2
asyncpg==0.26.0
attrs==22.1.0
awesome-slugify==1.6.5
beautifulsoup4==4.8.2
boto3==1.9.124
botocore==1.12.253
cachetools==4.2.4
camelot-py==0.10.1
cffi==1.15.1
chardet==3.0.4
charset-normalizer==2.1.0
click==8.1.3
cloudpickle==2.1.0
commonmark==0.9.1
coolname==1.1.0
coverage==6.4.3
croniter==1.3.5
cryptography==37.0.4
decorator==5.1.1
distro==1.7.0
docker==6.0.0
docutils==0.15.2
elasticsearch==7.13.1
et-xmlfile==1.1.0
Faker==13.15.1
fastapi==0.79.1
frozenlist==1.3.1
fsspec==2022.7.1
future==0.18.2
fuzzywuzzy==0.17.0
gcsfs==2021.4.0
ghostscript==0.7
google-api-core==2.8.2
google-auth==1.35.0
google-auth-oauthlib==0.5.2
google-cloud-core==1.7.3
google-cloud-storage==1.38.0
google-crc32c==1.3.0
google-resumable-media==1.3.3
googleapis-common-protos==1.56.4
greenlet==1.1.2
griffe==0.21.0
h11==0.12.0
hare==1.2.0
headless==72.0.3618.0+1.gfe69cf8
httpcore==0.15.0
httpx==0.23.0
idna==2.8
importlib-metadata==4.12.0
iniconfig==1.1.1
jmespath==0.10.0
jsonpatch==1.32
jsonpointer==2.3
keyring==23.8.2
keyrings.google-artifactregistry-auth==1.0.0
kubernetes==24.2.0
lxml==4.4.2
lzstring==1.0.4
Mako==1.2.1
MarkupSafe==2.1.1
multidict==6.0.2
nameparser==1.1.1
numpy==1.19.5
oauthlib==3.2.0
olefile==0.46
opencv-python==4.4.0.46
openpyxl==3.0.10
orderedmultidict==1.0.1
orjson==3.5.1
packaging==21.3
pandas==1.4.3
pathspec==0.9.0
pdfminer.six==20201018
pdftopng==0.2.3
pendulum==2.1.2
phonenumbers==8.12.52
pika==1.2.0
Pillow==4.0.0
pluggy==1.0.0
prefect==2.2.0
protobuf==4.21.5
py==1.11.0
pyarrow==9.0.0
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyclops==2.1.0
pycparser==2.21
pydantic==1.9.2
Pygments==2.13.0
pyocr==0.7.2
pyOpenSSL==22.0.0
pyparsing==3.0.9
PyPDF2==2.9.0
PySocks==1.7.1
pyssdb==0.4.2
pytest==7.1.2
pytest-cov==3.0.0
python-dateutil==2.8.2
python-dotenv==0.17.1
python-Levenshtein==0.12.0
python-slugify==6.1.2
pytz==2022.2.1
pytzdata==2020.1
pyxlsb==1.0.9
PyYAML==6.0
readchar==4.0.2
regex==2022.7.25
requests==2.28.1
requests-ftp==0.3.1
requests-oauthlib==1.3.1
requests-toolbelt==0.9.1
rfc3986==1.5.0
rich==12.5.1
rosetta==0.2.2
rsa==4.9
Rtree==0.9.3
s3transfer==0.2.1
selenium==3.3.1
six==1.16.0
slack-sdk==3.18.1
sniffio==1.2.0
sortedcontainers==2.4.0
soupsieve==2.3.2.post1
SQLAlchemy==1.4.40
starlette==0.19.1
stem==1.8.0
tabula-py==2.4.0
tabulate==0.8.10
tabulawrapper==2.0+5.gbf09143
text-unidecode==1.3
tika==1.23
toml==0.10.2
tomli==2.0.1
typer==0.6.1
typing_extensions==4.3.0
Unidecode==0.4.21
urllib3==1.26.12
uvicorn==0.18.2
websocket-client==1.3.3
xlrd==1.1.0
yarl==1.8.1
ziplus==1.0+8.ga022d63
zipp==3.8.1
I did also notice that the 
GCS
class in 
prefect.filesystems
is the only one that doesn't have 
_block_type_name
defined. Not sure if that has anything to do with this though.
r
Blocks fall back to using the class name if 
_block_type_name
isn't specified so I don't think that is it
k
That's what I assumed as soon as I posted that lol
it's gotta be a dependency conflict pip didn't catch given the flow works fine on the clean prefect image
doing a pip freeze on a brand new env with just prefect==2.2.0 installed and comparing against a pip freeze on my env, and noting the differences for prefect's reqs. will update again
my env left, fresh prefect install right
Copy code
cachetools==4.2.4 vs cachetools==5.2.0
charset-normalizer==2.1.0 vs charset-normalizer==2.1.1
fastapi==0.79.1 vs fastapi==0.81.0
greenlet==1.1.2 vs greenlet==1.1.3
idna==2.8 vs idna==3.3
websocket-client==1.3.3 vs websocket-client==1.4.0
though it appears some of these are a matter of when I made my existing local env
r
thanks, Kevin - I'm reaching out to some of my colleagues here to see if they know the answer to this
ā¤ļø 1
a
Kevin, can you get into a bash of the running container and investigate there whether you can import prefect and gcsfs fine and whether the versions are OK? Also, could you share how you created the deployment with this block? If nothing else works it would be helpful if you could create a GitHub issue with an MRE so that we can try and reproduce, and troubleshoot further if needed
k
hi anna, thanks for the advice here. I'm running the image in a local container to check these things. Prefect and gcsfs are both on their latest versions and import successfully in python. I can even write the same prefect flow in the container and run it successfully
I'll keep trying things and see what I can come up with
ahhhh I think I found it. prefect uses 
python-slugify
my project uses 
awesome-slugify
both import as 
slugify
I discovered this by trying to start an agent using my image, and the request failed because the slug in the body wasn't formatted correctly.
r
That sounds right, since the default behavior of blocks is to use a slugify-generated slug to register the class
k
yep. well that's fun.
I haven't run into a same-name package conflict like this before, so I'll do some thinking about how I can resolve it. Thanks for digging in with me, I really appreciate it
r
You're welcome! Thanks for the work you put in to track this down.
3 Views
Open in Slack
PreviousNext
Powered by