Hi Folks, when deploying self-hosted Prefect server on GCP, I am encountering this error: ```DEFAULT...

Ruslan Bergenov

09/06/2024, 11:31 PM

Hi Folks, when deploying self-hosted Prefect server on GCP, I am encountering this error:

Copy code

DEFAULT 2024-09-06T23:24:20.993197Z Warning! Failed to load collection 'prefect_gcp': ImportError: cannot import name 'SecretDict' from 'prefect.types' (/usr/local/lib/python3.10/site-packages/prefect/types/__init__.py)

I am using this Docker image:

Copy code

--image=prefecthq/prefect:2-latest

Anyone else has the same error? Any insight on fixes or workarounds?

Nate

09/07/2024, 12:37 AM

hmm this feels like a version mismatch to me as in, you might have a 3.x version of prefect-gcp but 2.x version of prefect are you installing an unbounded prefect-gcp via EXTRA_PIP_PACKAGES or somehow else at runtime?

Chris White

09/07/2024, 12:40 AM

+1 to what nate said -- you might try repulling the latest image if this image was pulled early this morning

Yanic Möller

09/10/2024, 9:06 AM

I have the same problem. The base image I use is

prefecthq/prefect:2.20.6-python3.9

and I explicitly install the following requirements for Prefect:

Copy code

prefect==2.20.6
prefect-docker==0.5.5
prefect-gcp==0.5.17

I've already tried changing to

prefect-gcp==0.5.8

but this returns the same error.

Yanic Möller

09/10/2024, 9:07 AM

@Ruslan Bergenov Did you find a solution for this?

Yanic Möller

09/10/2024, 1:09 PM

OK, I have figured this out https://prefect-community.slack.com/archives/C04DZJC94DC/p1725973734291069?thread_ts=1725959425.128269&cid=C04DZJC94DC

Yanic Möller

09/10/2024, 1:12 PM

IMO this is difficult to understand from the docs alone. It's not explicitly explained what the required package is.

Nate

09/10/2024, 1:15 PM

hi @Yanic Möller - glad you figured it out! i pinned a message about this in #CL09KU1K7 suggesting a a safe way to install extras, eg pip install ‘prefect[gcp]<3’ and the suggestion to use the extras is here in the docs perhaps we should update the integrations guide as well to contain this info

🙏 1

Ruslan Bergenov

09/10/2024, 4:54 PM

Thank you very much folks for your feedback. I haven’t had a chance to investigate it more yet.

👍 1

Ruslan Bergenov

09/13/2024, 11:44 PM

@Nate RE: _are you installing an unbounded prefect-gcp via EXTRA_PIP_PACKAGES or somehow else at runtime?_ I was installing it like this, while deploying a Cloud Run worker:

Copy code

gcloud run deploy $WORKER --image=prefecthq/prefect:2-latest \
--set-env-vars
........
--args "prefect","worker","start","--install-policy","always","--with-healthcheck","-p","cloud_run_work_pool","-t","cloud-run"

I will try the suggested workarounds.

Ruslan Bergenov

09/14/2024, 12:05 AM

I just deleted the 2 Cloud Run services (worker and server). Then I redeployed my worker and server on GCP as Cloud Run services from scratch and the error is gone. I am assuming the dependency error may have been fixed.

Copy code

gcloud run deploy $SERVER --image=prefecthq/prefect:2-latest
…..

Copy code

gcloud run deploy $WORKER --image=prefecthq/prefect:2-latest
….

46 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.