Loup
01/11/2024, 7:11 PMJake Kaplan
01/11/2024, 7:13 PMLoup
01/11/2024, 7:46 PMif __name__ == "__main__":
flow.from_source(
source="<https://github.com/lxup/recomend-prefect.git>",
entrypoint="flows/tmdb_update.py:tmdb_update",
).deploy(
name="tmdb_update",
work_pool_name="worker",
cron="0 9 * * *",
job_variables={
"env": {
"EXTRA_PIP_PACKAGES": "supabase more_itertools psycopg2-binary",
}
}
)
Loup
01/11/2024, 7:46 PMModuleNotFoundError: No module named 'more_itertools'
Loup
01/11/2024, 8:36 PMLoup
01/11/2024, 11:14 PMFROM prefecthq/prefect:latest
RUN pip install supabase more_itertools psycopg2-binary
But Im getting this error :
Flow run infrastructure exited with non-zero status code:
Task classic-sawfish-j4qs9-task0 failed with message: The container exited with an error. (Error Code: 1)
This may be caused by attempting to run an image with a misspecified platform or architecture.Will Raphaelson
01/11/2024, 11:58 PMWill Raphaelson
01/11/2024, 11:58 PMLoup
01/12/2024, 12:03 AMdocker build -t europe-west9-docker.pkg.dev/prefect-cloud-run/prefect-cloud/prefect:lastest .
docker push europe-west9-docker.pkg.dev/prefect-cloud-run/prefect-cloud/prefect:lastest
My Dockerfile is :
# We're using the latest version of Prefect with Python 3.10
FROM prefecthq/prefect:2-latest
# Add our requirements.txt file to the image and install dependencies
COPY requirements.txt .
RUN pip install -r requirements.txt --trusted-host pypi.python.org --no-cache-dir
Requirements.txt contain my differente pip lib :
supabase
more_itertools
psycopg2-binary
But my script is pretty heavy in term of power, so do Google Cloud Run gonna handle it ? Because right now im testing the Prefect Cloud Worker, and Im getting a memory rate limite :
Flow run infrastructure exited with non-zero status code:
Task flow-run-92c7e781-8b7a-4acf-951e-34c01625ef90-21b252ae51dcjhfqw-task0 failed with message: The configured memory limit was reached. (Error Code: 1)
Will Raphaelson
01/12/2024, 12:06 AM--platform linux/amd64
while i ask some folks internally?Loup
01/12/2024, 12:15 AMlinux/amd64
?Will Raphaelson
01/12/2024, 12:15 AMLoup
01/12/2024, 12:17 AMWill Raphaelson
01/12/2024, 12:18 AM--platform linux/amd64
and see if that runs?Will Raphaelson
01/12/2024, 12:22 AMLoup
01/12/2024, 12:50 AM--plateform
flag ?
Like that ?
docker build -t <http://europe-west9-docker.pkg.dev/prefect-cloud-run/prefect-cloud/prefect:lastest|europe-west9-docker.pkg.dev/prefect-cloud-run/prefect-cloud/prefect:lastest> . --platform linux/amd64
Will Raphaelson
01/12/2024, 12:50 AMWill Raphaelson
01/12/2024, 12:51 AMLoup
01/12/2024, 12:58 AMLoup
01/12/2024, 12:58 AMLoup
01/12/2024, 12:58 AMLoup
01/12/2024, 12:58 AMLoup
01/12/2024, 1:31 AMEncountered exception during execution:
Traceback (most recent call last):
...
02:02:45 AM
prefect.flow_runs
ERROR
Finished in state Failed('Flow run encountered an exception. MissingResult: The result was not persisted and is no longer available.')
Well Ive buy one month VPS with 4vCPU and 8GbRAM from Contabo to test and its workingLoup
01/12/2024, 1:32 AMLoup
01/12/2024, 1:44 AMWill Raphaelson
01/12/2024, 2:41 PMKevin Grismore
01/12/2024, 5:06 PMif __name__ == "__main__":
flow.from_source(
source="<https://github.com/lxup/recomend-prefect.git>",
entrypoint="flows/tmdb_update.py:tmdb_update",
).deploy(
name="tmdb_update",
work_pool_name="worker",
cron="0 9 * * *",
job_variables={
"memory": "8G",
}
)
Loup
01/12/2024, 5:50 PMKevin Grismore
01/12/2024, 5:59 PM