Prefect appears to be trying to launch my Dask cluster 3 times when I run my flow:

2022-09-29 11:50:53 - coiled - INFO - Using existing cluster: 'ghg-demo (id: 83299)'
2022-09-29 11:50:53 - coiled - INFO - Creating Cluster (name: ghg-demo, <https://cloud.coiled.io/abcd1234/clusters/83299/details> ). This might take a few minutes...
2022-09-29 11:50:54 - coiled - INFO - Scheduler: ready                               Workers: 3 ready (of 3)
2022-09-29 11:50:54 - coiled - INFO - Scheduler: ready                               Workers: 3 ready (of 3)
2022-09-29 11:50:59 - coiled - INFO - Using existing cluster: 'ghg-demo (id: 83299)'
2022-09-29 11:50:59 - coiled - INFO - Creating Cluster (name: ghg-demo, <https://cloud.coiled.io/abcd1234/clusters/83299/details> ). This might take a few minutes...
2022-09-29 11:50:59 - coiled - INFO - Scheduler: ready                               Workers: 3 ready (of 3)
2022-09-29 11:50:59 - coiled - INFO - Scheduler: ready                               Workers: 3 ready (of 3)
2022-09-29 11:51:05 - coiled - INFO - Using existing cluster: 'ghg-demo (id: 83299)'
2022-09-29 11:51:05 - coiled - INFO - Creating Cluster (name: ghg-demo, <https://cloud.coiled.io/abcd1234/clusters/83299/details> ). This might take a few minutes...
2022-09-29 11:51:06 - coiled - INFO - Scheduler: ready                               Workers: 3 ready (of 3)
2022-09-29 11:51:06 - coiled - INFO - Scheduler: ready                               Workers: 3 ready (of 3)

The simple flow is decorated as follows:

@flow(
    description = 'Calculates revenues for wells associated with a specific forecast',
    task_runner = CoiledTaskRunner(config_name=WELL_REVENUE_CONFIG)
)
def forecast_well_revenues(request: ForecastJobRequest) -> None:
    # invoke tasks here

Can someone explain why this is happening?

Prefect 1.0 -> 2.0 Question: Is there a reason that

Deployment.build_from_flow

requires a

.prefectignore

file? Currently have all of those flows in a pip installable python module, would be nice to be able to override that rather than need to skip upload in the build, and then separate calls to

apply()

and

upload_to_storage(...)

.

Hi There I'm having undeletable flow run https://app.prefect.cloud/account/a5aa4a76-f2a5-4e07-bb53-f0e7ff684f5b/workspace/f480[…]6-85a65419eb76/flow-run/82337252-1919-4165-b79a-717f313c4b24 is there anything to do? server is always replying with 500

Hello, just this afternoon I’ve started getting a 500 when trying to apply a deployment to my workspace (same deployment worked fine earlier) today)/ I can see there’s been quite a few changes around deployments causing issues with the 2.4.x releases, but is there any way to tell why it’s rejecting it? Output is just a generic 500 error.

i'm also getting 500 internal service errors when im trying to apply deployments

I am trying to run in process on a prefect agent on AKS loading data from azure storage. I am getting this log from the agent pod. I have deployed the pod from the prefect helm chart. I have set the env variable EXTRA_PIP_PACKAGES=adlfs on the pod. Is this error familiar? It clearly has something to do with storage. I wondered if the adlfs package was not being loaded.

2022-09-29T21:22:51.649459194Z 21:22:51.649 | INFO    | prefect.agent - Completed submission of flow run '78a45bb6-5ee3-4d6b-9a34-bf6ca56e496c'
2022-09-29T21:30:52.133263521Z 21:30:52.133 | INFO    | prefect.agent - Submitting flow run 'c716e857-eba0-432a-8561-3f622cff080a'
2022-09-29T21:30:52.859579518Z 21:30:52.859 | INFO    | prefect.infrastructure.process - Opening process 'scrupulous-collie'...
2022-09-29T21:30:52.860369014Z 21:30:52.859 | ERROR   | prefect.agent - Failed to submit flow run 'c716e857-eba0-432a-8561-3f622cff080a' to infrastructure.
2022-09-29T21:30:52.860389714Z Traceback (most recent call last):
2022-09-29T21:30:52.860399114Z   File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 233, in _submit_run_and_capture_errors
2022-09-29T21:30:52.860407714Z     result = await infrastructure.run(task_status=task_status)
2022-09-29T21:30:52.860461413Z   File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/process.py", line 71, in run
2022-09-29T21:30:52.860471313Z     with tempfile.TemporaryDirectory(suffix="prefect") as tmp_dir:
2022-09-29T21:30:52.860480013Z   File "/usr/local/lib/python3.10/tempfile.py", line 819, in __init__
2022-09-29T21:30:52.860488513Z     self.name = mkdtemp(suffix, prefix, dir)
2022-09-29T21:30:52.860497213Z   File "/usr/local/lib/python3.10/tempfile.py", line 357, in mkdtemp
2022-09-29T21:30:52.860526313Z     prefix, suffix, dir, output_type = _sanitize_params(prefix, suffix, dir)
2022-09-29T21:30:52.860534613Z   File "/usr/local/lib/python3.10/tempfile.py", line 126, in _sanitize_params
2022-09-29T21:30:52.860543013Z     dir = gettempdir()
2022-09-29T21:30:52.860551213Z   File "/usr/local/lib/python3.10/tempfile.py", line 299, in gettempdir
2022-09-29T21:30:52.860558913Z     return _os.fsdecode(_gettempdir())
2022-09-29T21:30:52.860567613Z   File "/usr/local/lib/python3.10/tempfile.py", line 292, in _gettempdir
2022-09-29T21:30:52.860575813Z     tempdir = _get_default_tempdir()
2022-09-29T21:30:52.860584713Z   File "/usr/local/lib/python3.10/tempfile.py", line 223, in _get_default_tempdir
2022-09-29T21:30:52.860615012Z     raise FileNotFoundError(_errno.ENOENT,
2022-09-29T21:30:52.860624512Z FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/opt/prefect']
2022-09-29T21:30:52.861818706Z 21:30:52.861 | INFO    | prefect.agent - Completed submission of flow run 'c716e857-eba0-432a-8561-3f622cff080a'

And now I have the error telling me to install adlfs even though it is in the EXTRA_PIP_PACKAGES env variable.

Traceback (most recent call last):
  File "/root/.Envs/healthgraph/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 442, in _prepare_and_execute
    self._rows = await prepared_stmt.fetch(*parameters)
  File "/root/.Envs/healthgraph/lib/python3.8/site-packages/asyncpg/prepared_stmt.py", line 176, in fetch
    data = await self.__bind_execute(args, 0, timeout)
  File "/root/.Envs/healthgraph/lib/python3.8/site-packages/asyncpg/prepared_stmt.py", line 241, in __bind_execute
    data, status, _ = await self.__do_execute(
  File "/root/.Envs/healthgraph/lib/python3.8/site-packages/asyncpg/prepared_stmt.py", line 230, in __do_execute
    return await executor(protocol)
  File "asyncpg/protocol/protocol.pyx", line 201, in bind_execute
asyncpg.exceptions.ActiveSQLTransactionError: ALTER TYPE ... ADD cannot run inside a transaction block

Coming across this error while trying to run a prefect agent and orion. Has anyone come across this before?

Hello, Is there backward compatibility for deployments ? Let's consider the following situations : 1 - Build deployment with prefect 2.0 => Run it with agent prefect 2.4 2 - Build deployment with prefect 2.4 => Run it with agent prefect 2.0 Can we do either of those? Or should we always build and run with the same version ? Because if if it's the latter, it will be really hard to upgrade to a different version in the future

Prefect helm chart results in following read_only error in the docker container running the prefect server Even when tweaking the config settings to run as root user. Is there any solution to this issue by tweaking the values.yaml? Edit: I've tried every possible combinations of # -- set orion containers' security context runAsNonRoot runAsNonRoot: true # -- set orion containers' security context readOnlyRootFilesystem readOnlyRootFilesystem: true # -- set orion containers' security context allowPrivilegeEscalation allowPrivilegeEscalation: false In both the pod and container security context It happens when i set the password for the postgressql subchart ## This is the password for

username

and will be set within the secret at the key

{fullnameOverride}-postgresql-password

. ## This argument is only relevant when using the Postgres database included in the chart. ## For an external postgres connection, you must create and use

existingSecret

instead of

postgresqlPassword

. # -- password for the custom user to create. Ignored if

auth.existingSecret

with key

password

is provided password: password

Hi everyone. We have not migrated to 2.0 because we use results in most of our flows and this feature is not available in 2.0. I understand results (and the ability to restart flows runs) is being worked on. 1) what's the target release date? 2) is there a current workaround (on local infrastructure and ECS), even if restarting is not currently possible? Thanks!

Hello all! I would like to set up a flow that pulls a GH repo of many other flows to execute them, similar to the dbt prefect flow of flows (minus the dbt) that @Anna Geller authored so well. In the repo, some flows should be executed daily, while others hourly. Does anyone have best practices on achieving this such that the parent flow runs on the hourly cadence and only picks up the hourly flows, while the daily ones get executed the one time? Further, for all child flows that are successful, how do they act as a potential dependency to trigger another flow, say a dbt task, without needing the whole flow to be successful?

Am seeing this exception while trying to run a simple flow:

File "/opt/conda/envs/coiled/lib/python3.9/site-packages/prefect/client/base.py", line 125, in raise_for_status
Exception: PrefectHTTPStatusError("Client error '404 Not Found' for url '<http://ephemeral-orion/api/task_runs/add94276-cb3e-417d-a3f9-033f935ae5ec/set_state>'\nResponse: {'exception_message': 'Task run with id add94276-cb3e-417d-a3f9-033f935ae5ec not found'}\nFor more information check: <https://httpstatuses.com/404>")

Any ideas as to what might cause this?

question: For using the default infastructure 'Process' how do i add environment variables to that in deployment?

Is there an issue with service accounts? The button on the ui is grayed out and inaccessible to create new SAs

I'm submitting a flow to Prefect Server for execution using

LocalRun

. I have a single Prefect agent running in a container. I use

IntervalSchedule

and set

interval=timedelta(seconds=60)

. I expected to see my flow run once every minute indefinitely, but instead, I see 10 flows scheduled on the Dashboard. I tried other schedulers -- Cron, etc -- but it always schedules 10 and only 10 runs. Am I missing something or misunderstanding how schedulers work? Thanks

hey all. Question about the

INTERNAL_SERVER_ERROR

and prefect's secret client throwing the

KeyError: 'SLACK_WEBHOOK_URL'

. I know there is a current intermittent issue in regards to retrieving secrets. We pass our secrets to the dask worker via the docker container so we aren't using the prefect secrets stored in the UI. However we are still getting this error and it has brought down multiple flows this morning. The error, other information and questions will be in the threads. Thank you in advance!

Hi folks - wondering if anyone can explain what exactly goes into storage? I'm running something relatively small-scale locally and my /.prefect/storage folder is starting to get quite bloated. I'm not running this through a deployment, so my understanding is that this isn't my whole codebase being stored as per the docs on storage. It seems implied (and would align with the files that I'm seeing) that individual task/flow result objects are getting saved here, but nothing in the prefect 2.0 docs actually say that explicitly - is that what's going on, and is there a way to manage them other than deleting/moving them manually?

Hi there! I was struggling to deploy a simple flow from code using

KubernetesJob

. However, just before posting a question in this channel, I figured I should "give it one last try" and to my surprise it actually worked 🤡 As the post is already written, I figured I might as well share was I experienced and the solutions so that anyone else having simiular issues may have somewhere it start. I was trying to create a deployment using

KubernetesJob

as infrastructure. The deployment looked like this (see code further down in the thread) The container image is built from a dockerfile based on

prefecthq/prefect:2-latest

and copies the script containing the flow to

/opt/prefect/flows/flow.py

. This deploys just fine, and I can see that the kubernetes job is created and the Prefect engine starts, however it does not seem like the flow runst, and I instead get the following error (see traceback further down in the thread) It turns out that the base docker-image I had chosen was the issue, as the prefect version did not match the version the server was running (

2.4.5

vs

2.4.2

). From the release-notes for

2.4.3

-

2.4.5

i see that block protection is an issue, and I speculate that this might be the root cause. Hope it helps 🙂 Edit: Moved code and traceback to the thread

Is there an expectation or best practice for where flows and supporting files are supposed to be to enable deploying?

Hi, I have been struggling to extract the logs for flow runs from prefect cloud using the api. In prefect 1 I was able to do this with the graph ql query below. How can I do this in prefect 2?

Hello, I am experiencing some strange behavior with Exceptions and I can't find anything in discourse about it. Exceptions being thrown from flow code are caught, but exceptions being thrown from libraries imported in the virtual environment are not being caught. Details in thread ...

Hello. Is there any best practice for handling large task results? I’ve got a task that can return >5GB of data as a result, and I assume that’s what’s causing the flow to slowdown creating and running subsequent tasks.

When will paused states be supported in Prefect 2.0? Currently this first class functionality is missing, which impacts our ability to migrate from 1.0

ents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Headers -arch arm64 -arch x86_64 -Werror=implicit-function-declaration -I/Users/navinchaganti/codebase/prefect-dev/prefect_env/include -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Headers -c asyncpg/pgproto/pgproto.c -o build/temp.macosx-10.9-universal2-cpython-39/asyncpg/pgproto/pgproto.o -O2 -fsigned-char -Wall -Wsign-compare -Wconversion
  asyncpg/pgproto/pgproto.c:31:10: fatal error: 'Python.h' file not found
  #include "Python.h"
           ^~~~~~~~~~
  1 error generated.
  error: command '/usr/bin/clang' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for asyncpg
Failed to build asyncpg
ERROR: Could not build wheels for asyncpg which use PEP 517 and cannot be installed directly

Anyone faced the asyncpg error while installation ?

Hello, I have some absolutely baffling behavior trying to implement state handlers using a

fail_signal = signals.FAIL(message_here); raise fail_signal

-> the state handler isn’t triggering, although it does trigger when a task otherwise changes status AND it triggers if a task fails for a reason other than a

raise fail_signal

. What even more baffling is that I can run the sample code with

mission_critical_task

from here and I get the expected behavior and the expected triggers. It’s a complicated flow, but I’ve ripped out as many pieces and simplified as much as possible and now I’m out of ideas on why I’m getting this lack of behavior. I’m tied to prefect 0.15.13

Hi everybody. When attempting to orchestrate dbt transformations via trigger_dbt_cli_command using an existing profile

~\.dbt\profiles.yml

, I receive the following runtime error and I'm having difficulty interpreting the issue,

exit code 4294967295

. Has anyone else countered this? Error:

"..~\venv\lib\site-packages\prefect_shell\commands.py", line 90, in shell_run_command raise RuntimeError(msg)

RuntimeError: Command failed with exit code 4294967295:

Script:

from prefect import flow
from prefect_dbt.cli.commands import trigger_dbt_cli_command

@flow
def trigger_dbt_cli_command_flow():
    result = trigger_dbt_cli_command("dbt debug")
    return result

trigger_dbt_cli_command_flow()

Versions: Windows 10 Prefect: 2.4.5 Prefect-dbt: 0.2.2 Thanks very much!

@Anna Geller I think I’m somewhat confuse. I’m using

awswrangler

package to wrangle data and move same to an S3 bucket. Before signing on prefect cloud I could use

wr

by simply indicating an s3 bucket path and my data would go to this path. Recently I created an S3 block via prefect cli. Now in an attempt to build deployment with the

-sb

flag I got an error that I indicated wrong s3 path. Clarifying questions: 1. Must I use the prefect cloud s3 storage block as the

path

under

wr

package? 2. Can I have a separate s3 bucket for storing output of my data wrangling different from the s3 bucket where my flow code should reside? I would appreciate if your answer include some basic scenarios. I have gone through all your repo and none appear to address this issue probably there is, maybe implicitly .