Tony Piazza
09/29/2022, 6:19 PM2022-09-29 11:50:53 - coiled - INFO - Using existing cluster: 'ghg-demo (id: 83299)'
2022-09-29 11:50:53 - coiled - INFO - Creating Cluster (name: ghg-demo, <https://cloud.coiled.io/abcd1234/clusters/83299/details> ). This might take a few minutes...
2022-09-29 11:50:54 - coiled - INFO - Scheduler: ready Workers: 3 ready (of 3)
2022-09-29 11:50:54 - coiled - INFO - Scheduler: ready Workers: 3 ready (of 3)
2022-09-29 11:50:59 - coiled - INFO - Using existing cluster: 'ghg-demo (id: 83299)'
2022-09-29 11:50:59 - coiled - INFO - Creating Cluster (name: ghg-demo, <https://cloud.coiled.io/abcd1234/clusters/83299/details> ). This might take a few minutes...
2022-09-29 11:50:59 - coiled - INFO - Scheduler: ready Workers: 3 ready (of 3)
2022-09-29 11:50:59 - coiled - INFO - Scheduler: ready Workers: 3 ready (of 3)
2022-09-29 11:51:05 - coiled - INFO - Using existing cluster: 'ghg-demo (id: 83299)'
2022-09-29 11:51:05 - coiled - INFO - Creating Cluster (name: ghg-demo, <https://cloud.coiled.io/abcd1234/clusters/83299/details> ). This might take a few minutes...
2022-09-29 11:51:06 - coiled - INFO - Scheduler: ready Workers: 3 ready (of 3)
2022-09-29 11:51:06 - coiled - INFO - Scheduler: ready Workers: 3 ready (of 3)
The simple flow is decorated as follows:
@flow(
description = 'Calculates revenues for wells associated with a specific forecast',
task_runner = CoiledTaskRunner(config_name=WELL_REVENUE_CONFIG)
)
def forecast_well_revenues(request: ForecastJobRequest) -> None:
# invoke tasks here
Can someone explain why this is happening?Joe
09/29/2022, 7:39 PMDeployment.build_from_flow
requires a .prefectignore
file? Currently have all of those flows in a pip installable python module, would be nice to be able to override that rather than need to skip upload in the build, and then separate calls to apply()
and upload_to_storage(...)
.Ahmed Ezzat
09/29/2022, 8:05 PMJosh Paulin
09/29/2022, 9:11 PMAlix Cook
09/29/2022, 9:40 PMJames Constable
09/29/2022, 11:06 PM2022-09-29T21:22:51.649459194Z 21:22:51.649 | INFO | prefect.agent - Completed submission of flow run '78a45bb6-5ee3-4d6b-9a34-bf6ca56e496c'
2022-09-29T21:30:52.133263521Z 21:30:52.133 | INFO | prefect.agent - Submitting flow run 'c716e857-eba0-432a-8561-3f622cff080a'
2022-09-29T21:30:52.859579518Z 21:30:52.859 | INFO | prefect.infrastructure.process - Opening process 'scrupulous-collie'...
2022-09-29T21:30:52.860369014Z 21:30:52.859 | ERROR | prefect.agent - Failed to submit flow run 'c716e857-eba0-432a-8561-3f622cff080a' to infrastructure.
2022-09-29T21:30:52.860389714Z Traceback (most recent call last):
2022-09-29T21:30:52.860399114Z File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 233, in _submit_run_and_capture_errors
2022-09-29T21:30:52.860407714Z result = await infrastructure.run(task_status=task_status)
2022-09-29T21:30:52.860461413Z File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/process.py", line 71, in run
2022-09-29T21:30:52.860471313Z with tempfile.TemporaryDirectory(suffix="prefect") as tmp_dir:
2022-09-29T21:30:52.860480013Z File "/usr/local/lib/python3.10/tempfile.py", line 819, in __init__
2022-09-29T21:30:52.860488513Z self.name = mkdtemp(suffix, prefix, dir)
2022-09-29T21:30:52.860497213Z File "/usr/local/lib/python3.10/tempfile.py", line 357, in mkdtemp
2022-09-29T21:30:52.860526313Z prefix, suffix, dir, output_type = _sanitize_params(prefix, suffix, dir)
2022-09-29T21:30:52.860534613Z File "/usr/local/lib/python3.10/tempfile.py", line 126, in _sanitize_params
2022-09-29T21:30:52.860543013Z dir = gettempdir()
2022-09-29T21:30:52.860551213Z File "/usr/local/lib/python3.10/tempfile.py", line 299, in gettempdir
2022-09-29T21:30:52.860558913Z return _os.fsdecode(_gettempdir())
2022-09-29T21:30:52.860567613Z File "/usr/local/lib/python3.10/tempfile.py", line 292, in _gettempdir
2022-09-29T21:30:52.860575813Z tempdir = _get_default_tempdir()
2022-09-29T21:30:52.860584713Z File "/usr/local/lib/python3.10/tempfile.py", line 223, in _get_default_tempdir
2022-09-29T21:30:52.860615012Z raise FileNotFoundError(_errno.ENOENT,
2022-09-29T21:30:52.860624512Z FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/opt/prefect']
2022-09-29T21:30:52.861818706Z 21:30:52.861 | INFO | prefect.agent - Completed submission of flow run 'c716e857-eba0-432a-8561-3f622cff080a'
James Constable
09/30/2022, 2:27 AMMalavika S Menon
09/30/2022, 10:06 AMTraceback (most recent call last):
File "/root/.Envs/healthgraph/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 442, in _prepare_and_execute
self._rows = await prepared_stmt.fetch(*parameters)
File "/root/.Envs/healthgraph/lib/python3.8/site-packages/asyncpg/prepared_stmt.py", line 176, in fetch
data = await self.__bind_execute(args, 0, timeout)
File "/root/.Envs/healthgraph/lib/python3.8/site-packages/asyncpg/prepared_stmt.py", line 241, in __bind_execute
data, status, _ = await self.__do_execute(
File "/root/.Envs/healthgraph/lib/python3.8/site-packages/asyncpg/prepared_stmt.py", line 230, in __do_execute
return await executor(protocol)
File "asyncpg/protocol/protocol.pyx", line 201, in bind_execute
asyncpg.exceptions.ActiveSQLTransactionError: ALTER TYPE ... ADD cannot run inside a transaction block
Coming across this error while trying to run a prefect agent and orion. Has anyone come across this before?Hamza Naanani
09/30/2022, 11:16 AMNic
09/30/2022, 11:30 AMusername
and will be set within the secret at the key {fullnameOverride}-postgresql-password
.
## This argument is only relevant when using the Postgres database included in the chart.
## For an external postgres connection, you must create and use existingSecret
instead of postgresqlPassword
.
# -- password for the custom user to create. Ignored if auth.existingSecret
with key password
is provided
password: passwordPedro Machado
09/30/2022, 12:30 PMSteven Trimboli
09/30/2022, 2:56 PMTony Piazza
09/30/2022, 3:11 PMFile "/opt/conda/envs/coiled/lib/python3.9/site-packages/prefect/client/base.py", line 125, in raise_for_status
Exception: PrefectHTTPStatusError("Client error '404 Not Found' for url '<http://ephemeral-orion/api/task_runs/add94276-cb3e-417d-a3f9-033f935ae5ec/set_state>'\nResponse: {'exception_message': 'Task run with id add94276-cb3e-417d-a3f9-033f935ae5ec not found'}\nFor more information check: <https://httpstatuses.com/404>")
Any ideas as to what might cause this?Taylor Babin
09/30/2022, 3:17 PMAlex Shea
09/30/2022, 4:15 PMYingbo
09/30/2022, 4:28 PMLocalRun
. I have a single Prefect agent running in a container. I use IntervalSchedule
and set interval=timedelta(seconds=60)
. I expected to see my flow run once every minute indefinitely, but instead, I see 10 flows scheduled on the Dashboard. I tried other schedulers -- Cron, etc -- but it always schedules 10 and only 10 runs. Am I missing something or misunderstanding how schedulers work? ThanksAndrew Pruchinski
09/30/2022, 4:46 PMINTERNAL_SERVER_ERROR
and prefect's secret client throwing the KeyError: 'SLACK_WEBHOOK_URL'
. I know there is a current intermittent issue in regards to retrieving secrets. We pass our secrets to the dask worker via the docker container so we aren't using the prefect secrets stored in the UI. However we are still getting this error and it has brought down multiple flows this morning. The error, other information and questions will be in the threads. Thank you in advance!Evan Dong
09/30/2022, 4:54 PMThomas Fredriksen
09/30/2022, 5:59 PMKubernetesJob
. However, just before posting a question in this channel, I figured I should "give it one last try" and to my surprise it actually worked 🤡
As the post is already written, I figured I might as well share was I experienced and the solutions so that anyone else having simiular issues may have somewhere it start.
I was trying to create a deployment using KubernetesJob
as infrastructure. The deployment looked like this (see code further down in the thread)
The container image is built from a dockerfile based on prefecthq/prefect:2-latest
and copies the script containing the flow to /opt/prefect/flows/flow.py
.
This deploys just fine, and I can see that the kubernetes job is created and the Prefect engine starts, however it does not seem like the flow runst, and I instead get the following error (see traceback further down in the thread)
It turns out that the base docker-image I had chosen was the issue, as the prefect version did not match the version the server was running (2.4.5
vs 2.4.2
). From the release-notes for 2.4.3
-2.4.5
i see that block protection is an issue, and I speculate that this might be the root cause.
Hope it helps 🙂
Edit: Moved code and traceback to the threadJoe
09/30/2022, 6:15 PMNick Coy
09/30/2022, 6:34 PMDavid Cupp
09/30/2022, 7:29 PMJosh Paulin
09/30/2022, 7:46 PMMatthew Seligson
09/30/2022, 8:20 PMNavin Chaganti
09/30/2022, 8:22 PMents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Headers -arch arm64 -arch x86_64 -Werror=implicit-function-declaration -I/Users/navinchaganti/codebase/prefect-dev/prefect_env/include -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Headers -c asyncpg/pgproto/pgproto.c -o build/temp.macosx-10.9-universal2-cpython-39/asyncpg/pgproto/pgproto.o -O2 -fsigned-char -Wall -Wsign-compare -Wconversion
asyncpg/pgproto/pgproto.c:31:10: fatal error: 'Python.h' file not found
#include "Python.h"
^~~~~~~~~~
1 error generated.
error: command '/usr/bin/clang' failed with exit code 1
----------------------------------------
ERROR: Failed building wheel for asyncpg
Failed to build asyncpg
ERROR: Could not build wheels for asyncpg which use PEP 517 and cannot be installed directly
Anyone faced the asyncpg error while installation ?Meghan Franklin
09/30/2022, 8:45 PMfail_signal = signals.FAIL(message_here); raise fail_signal
-> the state handler isn’t triggering, although it does trigger when a task otherwise changes status AND it triggers if a task fails for a reason other than a raise fail_signal
. What even more baffling is that I can run the sample code with mission_critical_task
from here and I get the expected behavior and the expected triggers. It’s a complicated flow, but I’ve ripped out as many pieces and simplified as much as possible and now I’m out of ideas on why I’m getting this lack of behavior. I’m tied to prefect 0.15.13Trevor Sweeney
09/30/2022, 10:34 PM~\.dbt\profiles.yml
, I receive the following runtime error and I'm having difficulty interpreting the issue, exit code 4294967295
. Has anyone else countered this?
Error:
"..~\venv\lib\site-packages\prefect_shell\commands.py", line 90, in shell_run_command raise RuntimeError(msg)
RuntimeError: Command failed with exit code 4294967295:
Script:
from prefect import flow
from prefect_dbt.cli.commands import trigger_dbt_cli_command
@flow
def trigger_dbt_cli_command_flow():
result = trigger_dbt_cli_command("dbt debug")
return result
trigger_dbt_cli_command_flow()
Versions:
Windows 10
Prefect: 2.4.5
Prefect-dbt: 0.2.2
Thanks very much!Miremad Aghili
09/30/2022, 10:36 PMapp.prefect.cloud
today. Is this maintenance time?Hedgar
10/01/2022, 8:17 AMawswrangler
package to wrangle data and move same to an S3 bucket. Before signing on prefect cloud I could use wr
by simply indicating an s3 bucket path and my data would go to this path. Recently I created an S3 block via prefect cli.
Now in an attempt to build deployment with the -sb
flag I got an error that I indicated wrong s3 path.
Clarifying questions:
1. Must I use the prefect cloud s3 storage block as the path
under wr
package?
2. Can I have a separate s3 bucket for storing output of my data wrangling different from the s3 bucket where my flow code should reside?
I would appreciate if your answer include some basic scenarios. I have gone through all your repo and none appear to address this issue probably there is, maybe implicitly .Bal Raj
10/02/2022, 8:53 AMFailed to load and execute flow run: ClientError([{'path': ['secret_value'], 'message': 'Unable to complete operation. An internal API error occurred.', 'extensions': {'code': 'API_ERROR'}}])
facing this issue multiple times? Any solutions?Bal Raj
10/02/2022, 8:53 AMFailed to load and execute flow run: ClientError([{'path': ['secret_value'], 'message': 'Unable to complete operation. An internal API error occurred.', 'extensions': {'code': 'API_ERROR'}}])
facing this issue multiple times? Any solutions?Anna Geller
10/02/2022, 12:51 PMTyson Chavarie
10/02/2022, 2:07 PMrequests.exceptions.ConnectionError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: / (Caused by ReadTimeoutError("HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Read timed out. (read timeout=15)"))
Failed to load and execute flow run: ClientError([{'path': ['secret_value'], 'message': 'Unable to complete operation. An internal API error occurred.', 'extensions': {'code': 'API_ERROR'}}])
Anna Geller
10/02/2022, 3:13 PMbrian
10/02/2022, 3:15 PM