Nimesh Kumar
02/21/2024, 11:28 AMWayne Lai
02/23/2024, 12:18 PMhow-to guide
, a push work pool was successfully provisioned. However, running the Prefect CLI generated example_deploy_script.py results the following error message, After restarting Docker Desktop on an Apple M2 machine, the error message remains the same. Any thoughts? Many thanks!
DockerException(docker.errors.DockerException: Error while fetching server API version: ('Connection aborted.', FileNotFoundError(2, 'No such file or directory')
Derek
02/24/2024, 5:46 PMNicholas Andonakis
02/26/2024, 12:49 AMSurya
02/27/2024, 1:51 AMNimesh Kumar
02/27/2024, 3:42 AMspehle
02/27/2024, 12:01 PMRicardo Schalch
02/27/2024, 8:17 PMversion: "3"
services:
server:
image: prefecthq/prefect:2-python3.11
container_name: prefect-server
restart: always
command: ["prefect", "server", "start"]
entrypoint: "/opt/prefect/entrypoint.sh"
ports:
- "4200:4200"
depends_on:
- postgres
environment:
PREFECT_DEBUG_MODE: "true"
PREFECT_API_DATABASE_CONNECTION_URL: '<postgresql+asyncpg://rbs:123@postgres/prefect>'
PREFECT_API_DATABASE_ECHO: "false"
PREFECT_API_DATABASE_MIGRATE_ON_START: "true"
postgres:
image: postgres:latest
container_name: prefect-db
restart: always
environment:
POSTGRES_PASSWORD: 123
POSTGRES_USER: rbs
POSTGRES_DB: prefect
ports:
- "5432:5432"
expose:
- 5432
volumes:
- db_data:/var/lib/postgresql/data
ui:
image: prefecthq/ui:latest
container_name: prefect-ui
restart: always
depends_on:
- server
ports:
- "8080:8080"
environment:
PREFECT_API_DATABASE_CONNECTION_URL: '<postgresql+asyncpg://rbs:123@postgres/prefect>'
PREFECT_API_DATABASE_ECHO: "false"
PREFECT_API_DATABASE_MIGRATE_ON_START: "true"
PREFECT_SERVER__APOLLO_URL: "/graphql"
PREFECT_API_URL: "/api"
PREFECT_SERVER__BASE_URL: "<http://localhost:4200>"
volumes:
db_data: {}
my main questions are:
1. why the server is not being accessible on my browser through localhost or 127.0.0.1 ... I'm using macos without any firewalls or anything like that. Prefect UI is being accessible through localhost 8080 though
2. I'm also confused where certain env variables goes and the right values for them.
thanks a lotWayne Lai
02/28/2024, 4:03 AMFile "/home/user/miniconda3/lib/python3.11/site-packages/prefect_docker/deployments/steps.py", line 334, in push_docker_image
username=credentials.get("username") AttributeError: 'str' object has no attribute 'get'
Here is the push clause in the yaml.
push:
- prefect_docker.deployments.steps.push_docker_image:
requires: prefect-docker>=0.3.1
image_name: '{{ build_image.image_name }}'
tag: '{{ build_image.tag }}'
credentials: '{{ prefect.blocks.docker-registry-credentials.dev-registry }'
Derek
02/28/2024, 2:15 PMBaker
02/28/2024, 10:02 PMfrom prefect.filesystems import S3
s3_block = S3.load("data-warehouse")
s3_block.read_path("test.txt")
>>> AttributeError: 'coroutine' object has no attribute 'read_path'
Andrew Rosen
03/01/2024, 7:49 PMMatthew Bell
03/03/2024, 11:19 PMlate
.
I feel like I'm missing an obvious step.aviv
03/04/2024, 8:41 AM.env
file to flow.run_config
is there something equivalent to that in prefect 2?Surya
03/05/2024, 6:15 AMNavaneethan
03/05/2024, 6:03 PMScript at 'gpt_review.py' encountered an exception: ModuleNotFoundError("No module named 'tqdm'")
See the below command, I ran prefect deployment command, it throws the error, when I import the module in python shell, it is imported. How to create prefect deployment with virtual env?
(myenv) navaneethan_ramasamy_turing_com@character-ai:/var/opt/repo/delivery-workflow$ prefect deployment build gpt_review.py:gpt_review --name=gpt_review
Script at 'gpt_review.py' encountered an exception: ModuleNotFoundError("No module named 'tqdm'")
(myenv) navaneethan_ramasamy_turing_com@character-ai:/var/opt/repo/delivery-workflow$ ipython
Python 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.22.2 -- An enhanced Interactive Python. Type '?' for help.
In [1]: import tqdm
In [2]:
Do you really want to exit ([y]/n)?
(myenv) navaneethan_ramasamy_turing_com@character-ai:/var/opt/repo/delivery-workflow$
James Morley
03/06/2024, 2:09 PMDaskTaskRunner
(specifying the cluster scheduler address) I get the following error:
Exception: PrefectHTTPStatusError("lient error '404 Not Found' for url '<http://ephemeral-prefect/api/task_runs/2832dc55-e6d8-4235-a6a5-ff4514b47067/set_state>
Based on https://docs.prefect.io/latest/concepts/infrastructure/:
ā¢ The ephemeral Prefect API won't work with Docker and Kubernetes. You must have a Prefect server or Prefect Cloud API endpoint set in your agent's configuration.The nodes on the SLURM cluster are using python via a Docker image. Is it the case that local Prefect execution just doesn't work with a remote Dask cluster and as per above I should use Prefect Cloud/Prefect server (was trying to start as simple as possible)?
Joey Allison
03/06/2024, 7:01 PM--limit
option. What does this do in practice? I have used the concurrency limit on work pools before, but I'm wondering what this achieves in addition to that.Joey Allison
03/06/2024, 11:16 PMWiebke Koepp
03/08/2024, 1:34 AMNimesh Kumar
03/08/2024, 5:23 AMJoĆ£o Vitor Bezerra
03/09/2024, 2:02 AMMatthew Bell
03/09/2024, 10:17 PMprefect.utilities.annotations.quote
shown below, as I've been getting a warning when passing a large dataframe to prefect tasks.
The docs mention the following:
> prefect.utilities.annotations.quote
will disable task dependency tracking for the wrapped object, but likely will increase performance
What exactly is task dependency tracking? Is that the little flow chart arrows shown in the prefect UI?Derek
03/10/2024, 3:52 PMMatthew Bell
03/10/2024, 7:02 PMprefect deploy
and following the CLI prompts. The CLI successfully pushes my image to dockerhub and successfully deploys my flow+deployment to my work pool on Prefect Cloud.
However, when triggering the deployment, Google Cloud Run fails to run it with the following error (images below)
The Prefect Cloud UI mentions:
Task transparent-platypus-fzwd9-task0 failed with message: The container exited with an error. (Error Code: 1)
This may be caused by attempting to run an image with a misspecified platform or architecture.This error is super vague, not really sure how to debug, but I imagine I'm making an obvious mistake here. Any idea how is it possible that this image runs locally with
docker run <image_name>
but fails to run on Google Cloud Run?
I've attached an image of my responses to the CLI prompts and my dockerfile (don't worry I will rotate that exposed API key)Joey Allison
03/13/2024, 5:56 PMActive Slots
and in the Cloud RestAPI an additional field called denied_slots
that are not described in the documentation. Could I ask for a little clarification of what those are and how to use them?Paul Gerard
03/15/2024, 10:10 AMMiguel Silva
03/15/2024, 10:20 PMdef trigger_dbt_flow() -> str:
result = DbtCoreOperation(
commands=["dbt run"],
project_dir="dbt_transforms", # dbt project directory
profiles_dir="dbt_transforms"
).run()
return result
John Utah
03/17/2024, 9:52 PM