https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • t

    Tony Piazza

    09/29/2022, 6:19 PM
    Prefect appears to be trying to launch my Dask cluster 3 times when I run my flow:
    2022-09-29 11:50:53 - coiled - INFO - Using existing cluster: 'ghg-demo (id: 83299)'
    2022-09-29 11:50:53 - coiled - INFO - Creating Cluster (name: ghg-demo, <https://cloud.coiled.io/abcd1234/clusters/83299/details> ). This might take a few minutes...
    2022-09-29 11:50:54 - coiled - INFO - Scheduler: ready                               Workers: 3 ready (of 3)
    2022-09-29 11:50:54 - coiled - INFO - Scheduler: ready                               Workers: 3 ready (of 3)
    2022-09-29 11:50:59 - coiled - INFO - Using existing cluster: 'ghg-demo (id: 83299)'
    2022-09-29 11:50:59 - coiled - INFO - Creating Cluster (name: ghg-demo, <https://cloud.coiled.io/abcd1234/clusters/83299/details> ). This might take a few minutes...
    2022-09-29 11:50:59 - coiled - INFO - Scheduler: ready                               Workers: 3 ready (of 3)
    2022-09-29 11:50:59 - coiled - INFO - Scheduler: ready                               Workers: 3 ready (of 3)
    2022-09-29 11:51:05 - coiled - INFO - Using existing cluster: 'ghg-demo (id: 83299)'
    2022-09-29 11:51:05 - coiled - INFO - Creating Cluster (name: ghg-demo, <https://cloud.coiled.io/abcd1234/clusters/83299/details> ). This might take a few minutes...
    2022-09-29 11:51:06 - coiled - INFO - Scheduler: ready                               Workers: 3 ready (of 3)
    2022-09-29 11:51:06 - coiled - INFO - Scheduler: ready                               Workers: 3 ready (of 3)
    The simple flow is decorated as follows:
    @flow(
        description = 'Calculates revenues for wells associated with a specific forecast',
        task_runner = CoiledTaskRunner(config_name=WELL_REVENUE_CONFIG)
    )
    def forecast_well_revenues(request: ForecastJobRequest) -> None:
        # invoke tasks here
    Can someone explain why this is happening?
    m
    a
    28 replies · 3 participants
  • j

    Joe

    09/29/2022, 7:39 PM
    Prefect 1.0 -> 2.0 Question: Is there a reason that
    Deployment.build_from_flow
    requires a
    .prefectignore
    file? Currently have all of those flows in a pip installable python module, would be nice to be able to override that rather than need to skip upload in the build, and then separate calls to
    apply()
    and
    upload_to_storage(...)
    .
    a
    5 replies · 2 participants
  • a

    Ahmed Ezzat

    09/29/2022, 8:05 PM
    Hi There I'm having undeletable flow run https://app.prefect.cloud/account/a5aa4a76-f2a5-4e07-bb53-f0e7ff684f5b/workspace/f480[…]6-85a65419eb76/flow-run/82337252-1919-4165-b79a-717f313c4b24 is there anything to do? server is always replying with 500
    ✅ 1
    s
    14 replies · 2 participants
  • j

    Josh Paulin

    09/29/2022, 9:11 PM
    Hello, just this afternoon I’ve started getting a 500 when trying to apply a deployment to my workspace (same deployment worked fine earlier) today)/ I can see there’s been quite a few changes around deployments causing issues with the 2.4.x releases, but is there any way to tell why it’s rejecting it? Output is just a generic 500 error.
    👀 1
    j
    7 replies · 2 participants
  • a

    Alix Cook

    09/29/2022, 9:40 PM
    i'm also getting 500 internal service errors when im trying to apply deployments
    👀 1
    j
    j
    13 replies · 3 participants
  • j

    James Constable

    09/29/2022, 11:06 PM
    I am trying to run in process on a prefect agent on AKS loading data from azure storage. I am getting this log from the agent pod. I have deployed the pod from the prefect helm chart. I have set the env variable EXTRA_PIP_PACKAGES=adlfs on the pod. Is this error familiar? It clearly has something to do with storage. I wondered if the adlfs package was not being loaded.
    2022-09-29T21:22:51.649459194Z 21:22:51.649 | INFO    | prefect.agent - Completed submission of flow run '78a45bb6-5ee3-4d6b-9a34-bf6ca56e496c'
    2022-09-29T21:30:52.133263521Z 21:30:52.133 | INFO    | prefect.agent - Submitting flow run 'c716e857-eba0-432a-8561-3f622cff080a'
    2022-09-29T21:30:52.859579518Z 21:30:52.859 | INFO    | prefect.infrastructure.process - Opening process 'scrupulous-collie'...
    2022-09-29T21:30:52.860369014Z 21:30:52.859 | ERROR   | prefect.agent - Failed to submit flow run 'c716e857-eba0-432a-8561-3f622cff080a' to infrastructure.
    2022-09-29T21:30:52.860389714Z Traceback (most recent call last):
    2022-09-29T21:30:52.860399114Z   File "/usr/local/lib/python3.10/site-packages/prefect/agent.py", line 233, in _submit_run_and_capture_errors
    2022-09-29T21:30:52.860407714Z     result = await infrastructure.run(task_status=task_status)
    2022-09-29T21:30:52.860461413Z   File "/usr/local/lib/python3.10/site-packages/prefect/infrastructure/process.py", line 71, in run
    2022-09-29T21:30:52.860471313Z     with tempfile.TemporaryDirectory(suffix="prefect") as tmp_dir:
    2022-09-29T21:30:52.860480013Z   File "/usr/local/lib/python3.10/tempfile.py", line 819, in __init__
    2022-09-29T21:30:52.860488513Z     self.name = mkdtemp(suffix, prefix, dir)
    2022-09-29T21:30:52.860497213Z   File "/usr/local/lib/python3.10/tempfile.py", line 357, in mkdtemp
    2022-09-29T21:30:52.860526313Z     prefix, suffix, dir, output_type = _sanitize_params(prefix, suffix, dir)
    2022-09-29T21:30:52.860534613Z   File "/usr/local/lib/python3.10/tempfile.py", line 126, in _sanitize_params
    2022-09-29T21:30:52.860543013Z     dir = gettempdir()
    2022-09-29T21:30:52.860551213Z   File "/usr/local/lib/python3.10/tempfile.py", line 299, in gettempdir
    2022-09-29T21:30:52.860558913Z     return _os.fsdecode(_gettempdir())
    2022-09-29T21:30:52.860567613Z   File "/usr/local/lib/python3.10/tempfile.py", line 292, in _gettempdir
    2022-09-29T21:30:52.860575813Z     tempdir = _get_default_tempdir()
    2022-09-29T21:30:52.860584713Z   File "/usr/local/lib/python3.10/tempfile.py", line 223, in _get_default_tempdir
    2022-09-29T21:30:52.860615012Z     raise FileNotFoundError(_errno.ENOENT,
    2022-09-29T21:30:52.860624512Z FileNotFoundError: [Errno 2] No usable temporary directory found in ['/tmp', '/var/tmp', '/usr/tmp', '/opt/prefect']
    2022-09-29T21:30:52.861818706Z 21:30:52.861 | INFO    | prefect.agent - Completed submission of flow run 'c716e857-eba0-432a-8561-3f622cff080a'
    ✅ 1
    k
    j
    6 replies · 3 participants
  • j

    James Constable

    09/30/2022, 2:27 AM
    And now I have the error telling me to install adlfs even though it is in the EXTRA_PIP_PACKAGES env variable.
    r
    c
    +1
    9 replies · 4 participants
  • m

    Malavika S Menon

    09/30/2022, 10:06 AM
    Traceback (most recent call last):
      File "/root/.Envs/healthgraph/lib/python3.8/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 442, in _prepare_and_execute
        self._rows = await prepared_stmt.fetch(*parameters)
      File "/root/.Envs/healthgraph/lib/python3.8/site-packages/asyncpg/prepared_stmt.py", line 176, in fetch
        data = await self.__bind_execute(args, 0, timeout)
      File "/root/.Envs/healthgraph/lib/python3.8/site-packages/asyncpg/prepared_stmt.py", line 241, in __bind_execute
        data, status, _ = await self.__do_execute(
      File "/root/.Envs/healthgraph/lib/python3.8/site-packages/asyncpg/prepared_stmt.py", line 230, in __do_execute
        return await executor(protocol)
      File "asyncpg/protocol/protocol.pyx", line 201, in bind_execute
    asyncpg.exceptions.ActiveSQLTransactionError: ALTER TYPE ... ADD cannot run inside a transaction block
    Coming across this error while trying to run a prefect agent and orion. Has anyone come across this before?
    b
    2 replies · 2 participants
  • h

    Hamza Naanani

    09/30/2022, 11:16 AM
    Hello, Is there backward compatibility for deployments ? Let's consider the following situations : 1 - Build deployment with prefect 2.0 => Run it with agent prefect 2.4 2 - Build deployment with prefect 2.4 => Run it with agent prefect 2.0 Can we do either of those? Or should we always build and run with the same version ? Because if if it's the latter, it will be really hard to upgrade to a different version in the future
    ✅ 1
    j
    4 replies · 2 participants
  • n

    Nic

    09/30/2022, 11:30 AM
    Prefect helm chart results in following read_only error in the docker container running the prefect server Even when tweaking the config settings to run as root user. Is there any solution to this issue by tweaking the values.yaml? Edit: I've tried every possible combinations of # -- set orion containers' security context runAsNonRoot runAsNonRoot: true # -- set orion containers' security context readOnlyRootFilesystem readOnlyRootFilesystem: true # -- set orion containers' security context allowPrivilegeEscalation allowPrivilegeEscalation: false In both the pod and container security context It happens when i set the password for the postgressql subchart ## This is the password for
    username
    and will be set within the secret at the key
    {fullnameOverride}-postgresql-password
    . ## This argument is only relevant when using the Postgres database included in the chart. ## For an external postgres connection, you must create and use
    existingSecret
    instead of
    postgresqlPassword
    . # -- password for the custom user to create. Ignored if
    auth.existingSecret
    with key
    password
    is provided password: password
    j
    6 replies · 2 participants
  • p

    Pedro Machado

    09/30/2022, 12:30 PM
    Hi everyone. We have not migrated to 2.0 because we use results in most of our flows and this feature is not available in 2.0. I understand results (and the ability to restart flows runs) is being worked on. 1) what's the target release date? 2) is there a current workaround (on local infrastructure and ECS), even if restarting is not currently possible? Thanks!
    ✅ 1
    🔜 1
    a
    2 replies · 2 participants
  • s

    Steven Trimboli

    09/30/2022, 2:56 PM
    Hello all! I would like to set up a flow that pulls a GH repo of many other flows to execute them, similar to the dbt prefect flow of flows (minus the dbt) that @Anna Geller authored so well. In the repo, some flows should be executed daily, while others hourly. Does anyone have best practices on achieving this such that the parent flow runs on the hourly cadence and only picks up the hourly flows, while the daily ones get executed the one time? Further, for all child flows that are successful, how do they act as a potential dependency to trigger another flow, say a dbt task, without needing the whole flow to be successful?
    ✅ 1
    a
    4 replies · 2 participants
  • t

    Tony Piazza

    09/30/2022, 3:11 PM
    Am seeing this exception while trying to run a simple flow:
    File "/opt/conda/envs/coiled/lib/python3.9/site-packages/prefect/client/base.py", line 125, in raise_for_status
    Exception: PrefectHTTPStatusError("Client error '404 Not Found' for url '<http://ephemeral-orion/api/task_runs/add94276-cb3e-417d-a3f9-033f935ae5ec/set_state>'\nResponse: {'exception_message': 'Task run with id add94276-cb3e-417d-a3f9-033f935ae5ec not found'}\nFor more information check: <https://httpstatuses.com/404>")
    Any ideas as to what might cause this?
    ✅ 1
    b
    m
    12 replies · 3 participants
  • t

    Taylor Babin

    09/30/2022, 3:17 PM
    question: For using the default infastructure 'Process' how do i add environment variables to that in deployment?
    ✅ 1
    k
    m
    28 replies · 3 participants
  • a

    Alex Shea

    09/30/2022, 4:15 PM
    Is there an issue with service accounts? The button on the ui is grayed out and inaccessible to create new SAs
    ✅ 1
    1 reply · 1 participant
  • y

    Yingbo

    09/30/2022, 4:28 PM
    I'm submitting a flow to Prefect Server for execution using
    LocalRun
    . I have a single Prefect agent running in a container. I use
    IntervalSchedule
    and set
    interval=timedelta(seconds=60)
    . I expected to see my flow run once every minute indefinitely, but instead, I see 10 flows scheduled on the Dashboard. I tried other schedulers -- Cron, etc -- but it always schedules 10 and only 10 runs. Am I missing something or misunderstanding how schedulers work? Thanks
    ✅ 1
    1️⃣ 1
    b
    7 replies · 2 participants
  • a

    Andrew Pruchinski

    09/30/2022, 4:46 PM
    hey all. Question about the
    INTERNAL_SERVER_ERROR
    and prefect's secret client throwing the
    KeyError: 'SLACK_WEBHOOK_URL'
    . I know there is a current intermittent issue in regards to retrieving secrets. We pass our secrets to the dask worker via the docker container so we aren't using the prefect secrets stored in the UI. However we are still getting this error and it has brought down multiple flows this morning. The error, other information and questions will be in the threads. Thank you in advance!
    1️⃣ 1
    ✅ 1
    j
    6 replies · 2 participants
  • e

    Evan Dong

    09/30/2022, 4:54 PM
    Hi folks - wondering if anyone can explain what exactly goes into storage? I'm running something relatively small-scale locally and my /.prefect/storage folder is starting to get quite bloated. I'm not running this through a deployment, so my understanding is that this isn't my whole codebase being stored as per the docs on storage. It seems implied (and would align with the files that I'm seeing) that individual task/flow result objects are getting saved here, but nothing in the prefect 2.0 docs actually say that explicitly - is that what's going on, and is there a way to manage them other than deleting/moving them manually?
    ✅ 1
    m
    4 replies · 2 participants
  • t

    Thomas Fredriksen

    09/30/2022, 5:59 PM
    Hi there! I was struggling to deploy a simple flow from code using
    KubernetesJob
    . However, just before posting a question in this channel, I figured I should "give it one last try" and to my surprise it actually worked 🤡 As the post is already written, I figured I might as well share was I experienced and the solutions so that anyone else having simiular issues may have somewhere it start. I was trying to create a deployment using
    KubernetesJob
    as infrastructure. The deployment looked like this (see code further down in the thread) The container image is built from a dockerfile based on
    prefecthq/prefect:2-latest
    and copies the script containing the flow to
    /opt/prefect/flows/flow.py
    . This deploys just fine, and I can see that the kubernetes job is created and the Prefect engine starts, however it does not seem like the flow runst, and I instead get the following error (see traceback further down in the thread) It turns out that the base docker-image I had chosen was the issue, as the prefect version did not match the version the server was running (
    2.4.5
    vs
    2.4.2
    ). From the release-notes for
    2.4.3
    -
    2.4.5
    i see that block protection is an issue, and I speculate that this might be the root cause. Hope it helps 🙂 Edit: Moved code and traceback to the thread
    :thank-you: 1
    ✅ 1
    m
    j
    4 replies · 3 participants
  • j

    Joe

    09/30/2022, 6:15 PM
    Is there an expectation or best practice for where flows and supporting files are supposed to be to enable deploying?
    1 reply · 1 participant
  • n

    Nick Coy

    09/30/2022, 6:34 PM
    Hi, I have been struggling to extract the logs for flow runs from prefect cloud using the api. In prefect 1 I was able to do this with the graph ql query below. How can I do this in prefect 2?
    ✅ 1
    c
    m
    5 replies · 3 participants
  • d

    David Cupp

    09/30/2022, 7:29 PM
    Hello, I am experiencing some strange behavior with Exceptions and I can't find anything in discourse about it. Exceptions being thrown from flow code are caught, but exceptions being thrown from libraries imported in the virtual environment are not being caught. Details in thread ...
    ✅ 1
    m
    10 replies · 2 participants
  • j

    Josh Paulin

    09/30/2022, 7:46 PM
    Hello. Is there any best practice for handling large task results? I’ve got a task that can return >5GB of data as a result, and I assume that’s what’s causing the flow to slowdown creating and running subsequent tasks.
    ✅ 1
    j
    m
    5 replies · 3 participants
  • m

    Matthew Seligson

    09/30/2022, 8:20 PM
    When will paused states be supported in Prefect 2.0? Currently this first class functionality is missing, which impacts our ability to migrate from 1.0
    ✅ 1
    j
    a
    8 replies · 3 participants
  • n

    Navin Chaganti

    09/30/2022, 8:22 PM
    ents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Headers -arch arm64 -arch x86_64 -Werror=implicit-function-declaration -I/Users/navinchaganti/codebase/prefect-dev/prefect_env/include -I/Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/Headers -c asyncpg/pgproto/pgproto.c -o build/temp.macosx-10.9-universal2-cpython-39/asyncpg/pgproto/pgproto.o -O2 -fsigned-char -Wall -Wsign-compare -Wconversion
      asyncpg/pgproto/pgproto.c:31:10: fatal error: 'Python.h' file not found
      #include "Python.h"
               ^~~~~~~~~~
      1 error generated.
      error: command '/usr/bin/clang' failed with exit code 1
      ----------------------------------------
      ERROR: Failed building wheel for asyncpg
    Failed to build asyncpg
    ERROR: Could not build wheels for asyncpg which use PEP 517 and cannot be installed directly
    Anyone faced the asyncpg error while installation ?
    ✅ 2
    m
    m
    25 replies · 3 participants
  • m

    Meghan Franklin

    09/30/2022, 8:45 PM
    Hello, I have some absolutely baffling behavior trying to implement state handlers using a
    fail_signal = signals.FAIL(message_here); raise fail_signal
    -> the state handler isn’t triggering, although it does trigger when a task otherwise changes status AND it triggers if a task fails for a reason other than a
    raise fail_signal
    . What even more baffling is that I can run the sample code with
    mission_critical_task
    from here and I get the expected behavior and the expected triggers. It’s a complicated flow, but I’ve ripped out as many pieces and simplified as much as possible and now I’m out of ideas on why I’m getting this lack of behavior. I’m tied to prefect 0.15.13
    k
    5 replies · 2 participants
  • t

    Trevor Sweeney

    09/30/2022, 10:34 PM
    Hi everybody. When attempting to orchestrate dbt transformations via trigger_dbt_cli_command using an existing profile
    ~\.dbt\profiles.yml
    , I receive the following runtime error and I'm having difficulty interpreting the issue,
    exit code 4294967295
    . Has anyone else countered this? Error:
    "..~\venv\lib\site-packages\prefect_shell\commands.py", line 90, in shell_run_command raise RuntimeError(msg)
    RuntimeError: Command failed with exit code 4294967295:
    Script:
    from prefect import flow
    from prefect_dbt.cli.commands import trigger_dbt_cli_command
    
    @flow
    def trigger_dbt_cli_command_flow():
        result = trigger_dbt_cli_command("dbt debug")
        return result
    
    trigger_dbt_cli_command_flow()
    Versions: Windows 10 Prefect: 2.4.5 Prefect-dbt: 0.2.2 Thanks very much!
    a
    j
    4 replies · 3 participants
  • m

    Miremad Aghili

    09/30/2022, 10:36 PM
    Hey Guys, I have not been able to access
    app.prefect.cloud
    today. Is this maintenance time?
    ✅ 1
    m
    t
    +1
    9 replies · 4 participants
  • h

    Hedgar

    10/01/2022, 8:17 AM
    @Anna Geller I think I’m somewhat confuse. I’m using
    awswrangler
    package to wrangle data and move same to an S3 bucket. Before signing on prefect cloud I could use
    wr
    by simply indicating an s3 bucket path and my data would go to this path. Recently I created an S3 block via prefect cli. Now in an attempt to build deployment with the
    -sb
    flag I got an error that I indicated wrong s3 path. Clarifying questions: 1. Must I use the prefect cloud s3 storage block as the
    path
    under
    wr
    package? 2. Can I have a separate s3 bucket for storing output of my data wrangling different from the s3 bucket where my flow code should reside? I would appreciate if your answer include some basic scenarios. I have gone through all your repo and none appear to address this issue probably there is, maybe implicitly .
    ✅ 1
    a
    3 replies · 2 participants
  • b

    Bal Raj

    10/02/2022, 8:53 AM
    Failed to load and execute flow run: ClientError([{'path': ['secret_value'], 'message': 'Unable to complete operation. An internal API error occurred.', 'extensions': {'code': 'API_ERROR'}}])
    facing this issue multiple times? Any solutions?
    a
    t
    +1
    6 replies · 4 participants
Powered by Linen
Title
b

Bal Raj

10/02/2022, 8:53 AM
Failed to load and execute flow run: ClientError([{'path': ['secret_value'], 'message': 'Unable to complete operation. An internal API error occurred.', 'extensions': {'code': 'API_ERROR'}}])
facing this issue multiple times? Any solutions?
a

Anna Geller

10/02/2022, 12:51 PM
thanks so much for reporting this! We have an open issue for this here and the last message I heard about it was that we are continuing to work on stabilizing the service backing the secrets API. For now, we deployed client-side retries on this kind of failure, so this shouldn't occur too frequently, LMK if you see different behavior. Thanks again for flagging and giving feedback on this
t

Tyson Chavarie

10/02/2022, 2:07 PM
I am also receiving a similar failure and do use prefect secrets. Is this error related?
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Max retries exceeded with url: / (Caused by ReadTimeoutError("HTTPSConnectionPool(host='<http://api.prefect.io|api.prefect.io>', port=443): Read timed out. (read timeout=15)"))
I am also seeing the same message as @Bal Raj on other runs:
Failed to load and execute flow run: ClientError([{'path': ['secret_value'], 'message': 'Unable to complete operation. An internal API error occurred.', 'extensions': {'code': 'API_ERROR'}}])
:thank-you: 1
a

Anna Geller

10/02/2022, 3:13 PM
I reported this to the platform team, we are working on stabilizing the secret retrieval service, thanks to both of you for reporting this
👍 1
b

brian

10/02/2022, 3:15 PM
I've also seen multiple failures with one of our prefect cloud flows this morning due to this issue. Thanks for relaying to the platform team @Anna Geller
:thank-you: 1
👍 1
View count: 3