https://prefect.io
Join Slack
Someone got a guide to setup a Google Cloud run for Prefect, im completly new with Google Cloud Run
# prefect-cloud
l
Someone got a guide to setup a Google Cloud run for Prefect, im completly new with Google Cloud Run
j
l
Okay i think I can reach my Clodu run instance. But Im not able to install additional pip library with my deploy :
Copy code
if __name__ == "__main__":
    flow.from_source(
    	source="<https://github.com/lxup/recomend-prefect.git>",
    	entrypoint="flows/tmdb_update.py:tmdb_update",
    ).deploy(
        name="tmdb_update",
        work_pool_name="worker",
        cron="0 9 * * *",
        job_variables={
            "env": {
                "EXTRA_PIP_PACKAGES": "supabase more_itertools psycopg2-binary",
            }
        }
    )
Im getting 
ModuleNotFoundError: No module named 'more_itertools'
I’m gonna try de make my own image
Ive created my own image like this : 
FROM prefecthq/prefect:latest
RUN pip install supabase more_itertools psycopg2-binary
But Im getting this error : Flow run infrastructure exited with non-zero status code: Task classic-sawfish-j4qs9-task0 failed with message: The container exited with an error. (Error Code: 1) This may be caused by attempting to run an image with a misspecified platform or architecture.
w
can you share what command you are using to build the image? the platform matters, there is a mention of it in the docs here
l
If I cant use the linux/amd64, which architecture im supposed to use 😅 ? There is my command to build the image :
Copy code
docker build -t europe-west9-docker.pkg.dev/prefect-cloud-run/prefect-cloud/prefect:lastest . 
docker push europe-west9-docker.pkg.dev/prefect-cloud-run/prefect-cloud/prefect:lastest
My Dockerfile is :
Copy code
# We're using the latest version of Prefect with Python 3.10
FROM prefecthq/prefect:2-latest

# Add our requirements.txt file to the image and install dependencies
COPY requirements.txt .
RUN pip install -r requirements.txt --trusted-host pypi.python.org --no-cache-dir
Requirements.txt contain my differente pip lib :
Copy code
supabase
more_itertools
psycopg2-binary
But my script is pretty heavy in term of power, so do Google Cloud Run gonna handle it ? Because right now im testing the Prefect Cloud Worker, and Im getting a memory rate limite :
Copy code
Flow run infrastructure exited with non-zero status code:
 Task flow-run-92c7e781-8b7a-4acf-951e-34c01625ef90-21b252ae51dcjhfqw-task0 failed with message: The configured memory limit was reached. (Error Code: 1)
w
im actually a little confused. Can you try specifically setting 
--platform linux/amd64
while i ask some folks internally?
l
Google Cloud Run isnt supposed to fail when running images built with 
linux/amd64
?
w
i am currently confused about that and trying to test it. I can update the docs if need be.
l
That weird because using the default prefect images with Google Cloud Run its working but my custom image isnt (I just add 3 pip install)
w
yeah im trying to figure this out as well. Can you try to build the image with 
--platform linux/amd64
and see if that runs?
ahh sorry - google cloud run needs amd64, the docs specify ar*m* is not supported not am*d.* I’m pretty confident if you build with the platform command i linked above thats going to work.
l
Im sorry but when Im suppod to qdd the 
--plateform
flag ? Like that ? 
docker build -t <http://europe-west9-docker.pkg.dev/prefect-cloud-run/prefect-cloud/prefect:lastest|europe-west9-docker.pkg.dev/prefect-cloud-run/prefect-cloud/prefect:lastest> . --platform linux/amd64
w
yep!
(also it says lastest, I think you mean latest)
l
wait its working
Im gonna try all the script to see
Ive just pay a month in other VPS ahah (Contabo for testing with 8gb of ram mb)
🤣😭
Well the script inside Google Cloud Run crash also probably because is to heavy in term of power needs. im getting this error :
Copy code
Encountered exception during execution:
Traceback (most recent call last):
...
02:02:45 AM
prefect.flow_runs
ERROR
Finished in state Failed('Flow run encountered an exception. MissingResult: The result was not persisted and is no longer available.')
Well Ive buy one month VPS with 4vCPU and 8GbRAM from Contabo to test and its working
So I guess there was an RAM issue
But anyway thx a lot @Will Raphaelson 🙏
w
Nice, we’re making progress.
k
you can request higher resources from cloud run for your deployment when you deploy. stealing your deployment script here:
Copy code
if __name__ == "__main__":
    flow.from_source(
    	source="<https://github.com/lxup/recomend-prefect.git>",
    	entrypoint="flows/tmdb_update.py:tmdb_update",
    ).deploy(
        name="tmdb_update",
        work_pool_name="worker",
        cron="0 9 * * *",
        job_variables={
            "memory": "8G",
        }
    )
l
Ohh didnt know that ! But its hard to have an idea oof the pricing of GOogle Cloud Run because my Contabo VPS is only 5.6e / months so cant know if Google gonna be cheaper but I should try
k
there is a decent pricing table here that you can use to estimate cost based on your resource requests + the runtime of your jobs
51 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by