https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • r

    Ray Bell

    09/16/2022, 4:41 PM
    I couldn't find a hello@prefect.com email to direct this at so apologies for dropping it here. I've been looped in to help with some last minute promotion/sponsorship for PyData Miami which is taking place next Thursday (9/22/2022): https://pydata.org/miami2022/ There is sponsorship opportunities if you think Prefect would be interested: https://pydata.org/miami2022/wp-content/uploads/2022/08/PyData-Miami-2022-Sponsorship-Prospectus-1.pdf You could also encourage attendance to your colleagues and network. Let me know if you have any questions.
    ✅ 1
    a
    1 reply · 2 participants
  • x

    Xavier Babu

    09/16/2022, 5:01 PM
    After installed Prefect Orion 2.4, getting the following error while starting Prefect Orion. Please help me to fix this issue. sqlalchemy.exc.ProgrammingError: (sqlalchemy.dialects.postgresql.asyncpg.ProgrammingError) <class 'asyncpg.exceptions.UndefinedObjectError'>: operator class "gin_trgm_ops" does not exist for access method "gin" [SQL: CREATE INDEX CONCURRENTLY trgm_ix_flow_name ON flow USING gin (name gin_trgm_ops); ]
    ✅ 2
    r
    15 replies · 2 participants
  • s

    Stephen Herron

    09/16/2022, 5:06 PM
    Do you know if it would be possible to convert the snowflake_utils to work with the SnowflakeConnector block? I’ve had a try but I get invalid username/password although I’ve used the same block in snowflake_query without an issue
    ✅ 1
    a
    2 replies · 2 participants
  • j

    Josh Paulin

    09/16/2022, 6:00 PM
    Hello. I’m starting to see a low of flows runs crashing when trying to update task state (500 error from https://api.prefect.cloud/api/accounts/<uuid>/workspaces/<uuid>/task_runs/<uuid>/set_state) Full stack trace in the thead. Is there any way to diagnose further?
    👀 2
    ✅ 1
    m
    v
    4 replies · 3 participants
  • n

    Nikhil Jain

    09/16/2022, 6:47 PM
    We are getting a flow failure with this message:
    Submitted for execution: Task arn:aws:ecs:us-west-2:<>:task/prod-prefect-cluster/c01a3ca13b3741a89e58b6c33dda0671
    INFO prefect-server.Lazarus.FlowRun Rescheduled by a Lazarus process. This is attempt 1.
    Any tips on how to debug this? I can’t find the logs for these tasks in cloudwatch.
    1️⃣ 1
    r
    1 reply · 2 participants
  • p

    Philip MacMenamin

    09/16/2022, 7:21 PM
    Question regarding trigger.any_failed:
    1️⃣ 1
    🙌 1
    c
    26 replies · 2 participants
  • c

    Chris Gunderson

    09/16/2022, 7:52 PM
    Hi Team - When running my docker image in the terminal. I wanted to get a list of block types: prefect block type ls There was an aws credential block type, but I am not able to create it, nor see that type in blocks. Is this a type that was abandoned?
    ✅ 1
    j
    5 replies · 2 participants
  • l

    Leon Kozlowski

    09/16/2022, 8:57 PM
    Is it expected that this graphql query would return a flow group id rather than an actual flow id? Does
    client.create_flow_run
    work off of flow group ids?
    {
      flow(
        where: {name: {_eq: "FLOW_NAME"}, archived: {_eq:false}},
        order_by: {created: desc}
      ) {
        id
        name
        project {
          name
        }
      }
    }
    ✅ 1
    b
    3 replies · 2 participants
  • s

    Sergiy Popovych

    09/16/2022, 9:27 PM
    The concept of Agent and Work Queues is really useful. But as far as I understand, Work Queue's hold only flows, not tasks. What queue do tasks go into? Is there a way for a user to tap into that queue?
    ✅ 1
    m
    r
    +1
    9 replies · 4 participants
  • a

    asaf alina

    09/17/2022, 1:21 PM
    Hi I have
    main
    flow, which calls to a subflow
    go_eat
    , each time with different parameters. something of the sort:
    @flow
    def go_eat():
        ...
    
    @flow
    def main():
        animals = ['dog', 'cat', 'monkey']
        for animal in animals:
            go_eat(animal)
    I want to make those
    go_eat
    flows to run in parallel, and I wonder how to implement this. it would be best if I could make up to 2 animals to
    go_eat
    in parallel, but no more than that. I'm not sure if I somehow better use the prefect queue concurrency mechanism, or if I better implement this in pure python. any suggestions?
    ✅ 1
    m
    t
    +1
    4 replies · 4 participants
  • y

    Yaron Levi

    09/18/2022, 4:12 AM
    Hi, We are getting an error when trying to login to Prefect Cloud 2.0:
    ✅ 1
    a
    7 replies · 2 participants
  • a

    Andrei Tulbure

    09/18/2022, 10:21 AM
    Hello Everyone. Quick questions. My prefect flow failed and the logs are not helping that much. I get a long stack trace error without anything specific. How can I know that I have a problem with the infra, or with the code ? If you need more details please help me frame my question better.
    ✅ 1
    a
    10 replies · 2 participants
  • j

    JP

    09/18/2022, 5:41 PM
    Heya! I’m new to prefect and am trying to deploy a local agent
    prefect deployment build flows/redshift_mv_refresh.py:redshift_mv_refresh --name JPMBP-O --queue JPMBP-O --tag "redshift mv-refresh test" --infra process
    However I’m running into an issue where the
    deployment build
    command tests the main flow and hence fails as it can not import some of the modules provided by a virtual env.
    ...
        import redshift_connector
    ModuleNotFoundError: No module named 'redshift_connector'
    ...
    prefect.exceptions.ScriptError: Script at 'flows/redshift_mv_refresh.py' encountered an exception
    An exception occurred.
    my project’s directory
    ❯ tree . -L 2
    .
    ├── flows
    │   ├── __pycache__
    │   ├── lib
    │   └── redshift_mv_refresh.py
    ├── requirements.txt
    └── venv
        ├── bin
        ├── include
        ├── lib
        └── pyvenv.cfg
    How can I tell
    deployment build
    to look for python in
    venv/bin/python
    Thank you a bunch! edit: I’m using Prefect 2 with Cloud.
    ✅ 3
    a
    3 replies · 2 participants
  • v

    Vadym Dytyniak

    09/19/2022, 8:25 AM
    Hi. What is the replacement for projects in Prefect 2? How to group flows? Thanks.
    ✅ 1
    a
    4 replies · 2 participants
  • d

    Dimosthenis Schizas

    09/19/2022, 10:28 AM
    Hello everyone, I'm looking to migrate from 1.0 to 2.0 and it seems that some functionality is missing. Back in 1.0 we were running a k8s agent and the flows where stored in a private Github repo. In Prefect 2.0, I wasn't able to find a way to connect to a private Github repo. I tried also to store the flows in GCS, however the agent didn't had the required library to connect and download the flows from it. Any chance I'm missing something?
    ✅ 1
    m
    j
    2 replies · 3 participants
  • j

    JP

    09/19/2022, 11:50 AM
    Is it possible to provide a list object as a parameter in Prefect Cloud 2 offering? When I input
    ['analytics.mv_1', 'analytics.mv_2']
    in the Parameters form, the parameter is treated as a string 😞
    ...
    13:31:26.728 | INFO    | Flow run 'lush-bettong' - Scheduling materialized views {mat_view_names}
    13:31:26.729 | INFO    | Flow run 'lush-bettong' - Refreshing materialized view '['
    13:31:26.735 | INFO    | Flow run 'lush-bettong' - Refreshing materialized view '''
    13:31:26.737 | INFO    | Flow run 'lush-bettong' - Refreshing materialized view 'a'
    13:31:26.739 | INFO    | Flow run 'lush-bettong' - Refreshing materialized view 'n'
    13:31:26.741 | INFO    | Flow run 'lush-bettong' - Refreshing materialized view 'a'
    13:31:26.745 | INFO    | Flow run 'lush-bettong' - Refreshing materialized view 'l'
    ...
    ✅ 1
    r
    2 replies · 2 participants
  • v

    Vlad Tudor

    09/19/2022, 12:22 PM
    Hi is there any Guide for the AWS setup necessary for a Prefect
    ECSAgent
    ? I don't quite manage to get it right. Thank you! The Agent registration is successful, I start the Flow, in AWS the task is created but the TaskDefinition is INACTIVE and the Task itself is PENDING
    ✅ 1
    r
    6 replies · 2 participants
  • g

    Giorgio Basile

    09/19/2022, 12:43 PM
    Hi, I was trying to find out in the Prefect 2.0 docs the equivalent of Prefect 1.0's context, as I would like to get the scheduled start time of the flow run inside the flow itself (or one of its tasks). Is it possible to do it? Thanks!
    ✅ 1
    s
    2 replies · 2 participants
  • b

    Bal Raj

    09/19/2022, 1:18 PM
    Hi, i was trying to trigger two independent flows (say flow1,flow2)from a parent flow while passing params(dict) from parent to flow2...flow2 is sorting the dict param received from parent..Can someone clear this out?Thanks. the later image shows the received param
    1️⃣ 1
    j
    3 replies · 2 participants
  • c

    Carlos Cueto

    09/19/2022, 1:29 PM
    Hello. I have a
    WebHookAction
    Automation set up that keeps generating false failure notifications on Cloud UI. The WebHook sends an HTTP POST to a specific endpoint and returns an HTTP 202 status code upon success, but Prefect seems to think it's an error. Every time the action triggers, I get a new failure notification. This literally spams our Cloud Notifications with false failures. FYI this issue seems to be the exact same issue, but I was told it wasn't, and to open a new ticket (I still think it's the same issue): https://github.com/PrefectHQ/prefect/issues/5955 Screenshot of false failure notification:
    ✅ 1
    r
    4 replies · 2 participants
  • j

    JP

    09/19/2022, 1:39 PM
    Hi again! Getting busy using Prefect 2 🙂 I’ve never worked with Prefect 1, but looking at the screenshots, there is an Agent tab available.. Do we have something like this in Prefect 2 UI? Might be subjective opinion, but anyway.. looking at Prefect 1 UI screenshots, it looks much more fleshed out than Prefect 2 UI. To your knowledge is the Prefect 2 UI work in progress and hence should I focus on using Prefect 1 for production use?
    ✅ 1
    o
    3 replies · 2 participants
  • a

    Adrien Besnard

    09/19/2022, 1:51 PM
    Hello! I was wondering: in Prefect 2, is it possible to leverage on multiple deployments to pass different parameters to the same flow?
    ✅ 1
    s
    4 replies · 2 participants
  • j

    Justin Trautmann

    09/19/2022, 2:02 PM
    hello prefect team, hello community, i am currently setting up a deployment for a flow with a couple of boolean parameters with default values in prefect 2 and noticed that these are nicely displayed as switches in the flow run creation form. The switches are set to "False" irrespective of the actual default value. When these switches are not changed during the flow run creation, the flow run will be submitted without these parameters, and consequently using the default settings that could be "True". This results in a mismatch between what the user sees on flow submission and what's actually executed. However, if the switches are toggled back and forth, effectively resulting in the same state of the form, the flows are submitted with the parameters from the form, overwriting the default parameters. This means that toggling the switches back and forth results in a different outcome than not touching the switches at all, even though the form looks exactly the same. i think this could lead to some very hard to reproduce issues. where should i submit UI related bug reports? Please find an example flow in thread. Thanks a lot
    👍 1
    ✅ 1
    c
    4 replies · 2 participants
  • a

    Alix Cook

    09/19/2022, 3:09 PM
    With prefect 2, I'm seeing issues when attempting to get flow groups via the api (using the cloud product). If I try to filter by flow group state, I get a 500 response, but if i omit that the request is fine.
    ✅ 1
    b
    m
    14 replies · 3 participants
  • j

    José Duarte

    09/19/2022, 4:03 PM
    Hey all, when the UI queries the API and fails, there is no indication in the UI, even a small badge would be helpful. I think this has been raised on GitHub but I am not sure. Does someone know if this is on the future plans?
    ✅ 1
    j
    j
    9 replies · 3 participants
  • n

    Nick Coy

    09/19/2022, 4:12 PM
    Hello, I cannot seem to run more than one k8 job at a time. If I try to kick off two jobs the first runs but the second job fails with a 500 error.
    ✅ 1
    n
    6 replies · 2 participants
  • n

    Nic

    09/19/2022, 4:30 PM
    I'm currently trying to implement Docker with prefect, but following Anna gellers repo with dockerfile, the docker build fails on both the copy and add line (2 and 4) anyone had the same issues or ideas of how to resolve it?
    ✅ 1
    j
    n
    11 replies · 3 participants
  • t

    Taylor Babin

    09/19/2022, 4:49 PM
    Hello all. I've been trying to deploy prefect 2.0 agent to an ec2 instance and been having issues. I tried following this page however it did not work. https://discourse.prefect.io/t/how-to-deploy-a-prefect-2-0-agent-to-an-ec2-instance-as-your-execution-layer/551 I was able to get the agent to run in the ec2 instance but when I would run a deployment it was just stay stuck on pending. The agent would pick up the flow and submit flow run but wouldn't run. Any help on how to get the agent running will be appreciated. Thank you, Taylor
    t
    16 replies · 2 participants
  • s

    Sam Garvis

    09/19/2022, 5:20 PM
    I was just curious why all these jobs stay around with the helm prefect-agent. I am using the most recent version of prefect-agentalso. Cuz I can't view the logs for previous jobs, it says
    Stream closed EOF for prod-prefect/analytic-shrimp5rkr9-j82lj (prefect-job)
    for all of them. Are they supposed to auto delete?
    ✅ 1
    k
    8 replies · 2 participants
  • j

    JP

    09/19/2022, 6:38 PM
    Heya! I’m developing a flow to run a bunch of materialised view refreshes with Prefect 2 (local agent + Cloud). These flows must never backfill as doing so yields no benefit. How can I tell my deployment to never backfill scheduled runs or is this something I should handle within the flow by checking the state (i.e. “late”) and have it report a successful flow run without running any tasks? It would be absolutely amazing, if someone can point me to a recipe or demo of any kind that I can lean on 🤷
    ✅ 1
    s
    m
    14 replies · 3 participants
Powered by Linen
Title
j

JP

09/19/2022, 6:38 PM
Heya! I’m developing a flow to run a bunch of materialised view refreshes with Prefect 2 (local agent + Cloud). These flows must never backfill as doing so yields no benefit. How can I tell my deployment to never backfill scheduled runs or is this something I should handle within the flow by checking the state (i.e. “late”) and have it report a successful flow run without running any tasks? It would be absolutely amazing, if someone can point me to a recipe or demo of any kind that I can lean on 🤷
✅ 1
s

Sam Garvis

09/19/2022, 6:51 PM
Wdym by backfill
And you're using Postgres?
j

JP

09/19/2022, 6:55 PM
I’ve shut down my agent. Below is how the queue looks like at the moment. If I start the agent now, it will try to run all “Late” jobs, essentially performing needless refreshes. I’d like it to either • Only run next scheduled • Run last “late” run and forget about all previous ones I strongly suspect this is something I need to handle within the flow, but not sure where to start 😞
m

Mason Menges

09/19/2022, 7:00 PM
Hey @JP I actually think an easier way to manage this might be to limit the number of scheduled runs that prefect creates, i.e. by defaults it attempts to schedule out 100 runs up to 100 days in the future, you can limit this in the configuration to just 1 preventing a backlog of flow runs from building up https://docs.prefect.io/concepts/schedules/.
j

JP

09/19/2022, 7:01 PM
Oh! I had no idea this option existed! I’ll look into this ASAP. Thank you @Mason Menges! 🙏
m

Mason Menges

09/19/2022, 7:02 PM
No problem 😄 Admittedly this can depend on how many runs you'd want to schedule in the same day but it might be worthing looking into, from there you could also implement some logic in your flow to check the scheduled start date for the run and compare that to the current time, if it falls out side of some predetermined range you could skip the rest of the tasks in the flow.
j

JP

09/19/2022, 7:09 PM
I see the 100 day / 100 runs default is for the scheduler service itself and not for deployment 😞 I’m looking for a solution per deployment or within a flow. Also since I’m using Prefect 2 (Cloud), I don’t think these scheduler service settings are available to me for alter.
I feel like I may have asked the question the wrong way.. let me rephrase ☀️ How can I tell my deployment / flow to not run any late schedules from the queue, only upcoming. I’m thinking retrieving
expected_start_time
(A) from flow context and compare it with
current time - 5 minutes
(B), if A < B, then have flow complete without running any tasks. There are couple of issues with my idea and it feels fairly hacky 😞 What would be the recommended approach to achieve something like this?
✅ 1
m

Mason Menges

09/20/2022, 3:41 PM
I don't believe there's a way to control this from the deployment or within the UI. Implementing some logic around the expected start time as you suggested would definitely be one way to accomplish this. All that said I think Cacheing the task run states would also be a viable option, essentially this wouldn't prevent the flow runs from triggering however you'd be able to persist the results of your task runs so that when the flow triggers again it just uses the cached state of the tasks instead of recomputing anything for some predetermined amount of time. https://docs.prefect.io/concepts/tasks/#task-arguments https://discourse.prefect.io/t/how-can-i-cache-a-task-result-for-two-hours-to-prevent-re-computation/67
j

JP

09/20/2022, 3:53 PM
Thank you, @Mason Menges! I’ll look into cache as well. I’ve attempted to retrieve some of the context metadata, which sort of works for some keys, but not others..
from prefect import flow, context

@flow(name="Hello Flow")
def hello_world():
    
    ctx = context.get_run_context()
    print('================')
    print('----CONTEXT----')
    print(ctx)
    print('----CONTEXT, start_time ----')
    print(ctx.get().start_time)
    print('----CONTEXT, expected_start_time ----')
    print(ctx.get().expected_start_time)
    print('================')

if __name__ == '__main__':
    hello_world()
I found this out solely by experimentation. I suspect I’m querying the flow’s context incorrectly with
ctx.get().expected_start_time
as it produces following error (full run in attachment)
...
AttributeError: 'FlowRunContext' object has no attribute 'expected_start_time'
...
If you have any docs on mechanics of Context object for prefect 2, please do point me in the correct direction 🙏
Untitled.txt
m

Mason Menges

09/20/2022, 4:10 PM
No problem 😄 You should be able to access the context for the flow run this way
from prefect import context

test_context = context.FlowRunContext.get()

start_time = test_context.flow_run.expected_start_time
As for docs this is a good place to reference https://docs.prefect.io/api-ref/prefect/context/
j

JP

09/20/2022, 4:14 PM
AH! I’m need new glasses 😄 It is in the
flow_run=FlowRun(..., estimated_run_time=foo)
. Thank you, thank you, thank you!
m

Mason Menges

09/20/2022, 4:32 PM
No problem at all 😄
View count: 3