https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • e

    Emma Rizzi

    12/10/2021, 10:43 AM
    Hi! I'm trying to create a custom VM image that launches a prefect agent on spawn, is there any records of such work already done here ? I haven't found anything but still worth to ask My goal is to simulate an elastic cluster with prefect: have a flow that launches a instance with this image, start a new sub-flow with this new agent, and deprovision the instance Almost everystep is validated, I just need to build the VM image in
    raw
    format and I got lost here, if anyone has experience with it I'd gladly take some advice!
    a
    a
    7 replies · 3 participants
  • s

    Suresh R

    12/10/2021, 11:11 AM
    Hi! I'm trying to find option similar to
    depends_on_past
    in airflow. We wanted next run of flow to start only when previous run is in Success state. Can anyone help?
    a
    22 replies · 2 participants
  • h

    haf

    12/10/2021, 11:35 AM
    Has anyone got spurious CancelledError:s when running with Dask? Flow id:
    e2631f0e-2987-405f-bf3b-3fbb46acf90c
    a
    a
    11 replies · 3 participants
  • r

    Ryan Brennan

    12/10/2021, 1:55 PM
    Hi, is there a way get the data that is used in the Dashboard UI via api calls? I’m mainly interested in recreating the Run History graph. For context, we’re an agency that implements/manages a bunch of Prefect instances and I would like to build a way for us to get daily summaries of our Client’s flow runs without needing to log in and out of the UI for each instance.
    a
    1 reply · 2 participants
  • f

    FuETL

    12/10/2021, 1:57 PM
    Hey guys, i'm struggling for a bit here. I need to define a custom
    context
    value inside the flow run (So i can retrieve later via graphql query), the issue is (not all my flow will have this custom context value), for what i understand the context values is passed in
    run()
    call, is not possible to add/modify and retrieve this values later? If so, there a alternative i can use to get custom values from a flow?
    a
    9 replies · 2 participants
  • d

    Daniil Ponizov

    12/10/2021, 2:49 PM
    I already ask this question in thread, but didnt get answer. I get this error, when trying to set the api with the --api argument at prefect agent start:
    line 831, in _verify_token
      raise AuthorizationError("No agent API token provided.")
    prefect.exceptions.AuthorizationError: No agent API token provided.
    a
    2 replies · 2 participants
  • s

    Stephen Herron

    12/10/2021, 3:49 PM
    hi - our security folks would like to understand the oauth grants for the slack app, before installing it into our workspace. Is this information readily available?
    k
    3 replies · 2 participants
  • a

    alex

    12/10/2021, 4:55 PM
    Hello, in which cases do I need to redeploy my flow on prefect? For example, if I have a task
    import app.logic
    @task
    def do_something():
         app.logic.dosomething()
    If I am using a Local Agent and modify
    app.logic.dosomething()
    without modifying the task or the general flow structure, would I still need to redeploy my flow?
    a
    1 reply · 2 participants
  • s

    Scott Hood

    12/10/2021, 5:21 PM
    Hey all! Just learning about Prefect, really cool stuff so far! I have primarily been looking into what the current solutions are for event-driven flows, and looking through the comments and documentation seems like in Prefect Core there isn't anything built in like Airflow Sensors (I have seen some of the AWS Lambda recommendations and blog posts). Looking at the Orion site however looks like this is being added with the newest engine? Is there any documentation regarding that or is that more so a feature coming soon to Orion and not yet hashed out?
    a
    8 replies · 2 participants
  • k

    Kevin Kho

    12/10/2021, 5:32 PM
    Join our livestream!
    :upvote: 1
  • s

    Shaoyi Zhang

    12/10/2021, 5:33 PM
    Hey team, I assume Prefect has nothing to do with Java, but double checking: Is prefect affected by this log4j vulnerability? https://www.bleepingcomputer.com/news/security/new-zero-day-exploit-for-log4j-java-library-is-an-enterprise-nightmare/
    k
    2 replies · 2 participants
  • m

    Martim Lobao

    12/10/2021, 5:37 PM
    question about manually setting a task from e.g. failed to success, is it possible to provide the output of the task?
    k
    7 replies · 2 participants
  • d

    dammy arinde

    12/10/2021, 5:43 PM
    Happy Friday! Please is there an example I can follow on using great expectations validation in prefect? I have it set up but when I run the flow, I get the error "ConfigNotFoundError('Error: No great_expectations directory was found here!" not sure what config I'm missing and where to add it. Thank you
    a
    a
    17 replies · 3 participants
  • j

    justabill

    12/10/2021, 7:00 PM
    👋 Hey community! Yesterday, we shipped some exciting improvements to Orion in the 2.0a6 release. Here are a couple that we're especially excited about: Subflow Navigation In the previous release, we introduced the ability to navigate seamlessly between subflows and parent flows via Radar :radar: . With this release, we made subflow navigation even easier by: - Enabling the dedicated subflow runs tab on the Flow Run page - Tracking of upstream inputs to subflow runs - Adding a flow and task run count to all subflow run cards in the Radar view - Adding a mini Radar view on the Flow run page Executors are now TaskRunners Previous versions of Prefect could only trigger execution of code defined within tasks. Orion can trigger execution of significant code that can be run outside of tasks. In order to make the role previously played by Prefect's
    Executor
    more explicit, we have renamed
    Executor
    to
    TaskRunner
    🏃. Check out the release notes to learn more about the other enhancements and fixes.
    🚀 2
    🎉 1
  • l

    Leon Kozlowski

    12/10/2021, 7:47 PM
    Hi all - pricing question - do child tasks in mapped task count as task runs? or does this apply to only the parent task?
    a
    k
    3 replies · 3 participants
  • t

    Tilak Maddy

    12/11/2021, 11:34 AM
    Hey everyone I have a flow that had required parameters. And I have a problem... I want to be able to programatically add schedule clocks with default parameters. I am able to do it through UI, however when it comes to the graphql solution ,[https://docs.prefect.io/orchestration/flow-runs/scheduling.html#creating-flow-schedules ] the biggest obstruction is that Setting flow group schedules will remove any existing schedules. I would like to have
    add_flow_group_schedule
    instead of just
    set_flow_group_schedule
    (Just like we have in the UI ~ The New Schedule button) -> How does it manage to do it ?
    a
    k
    14 replies · 3 participants
  • a

    Alvaro Durán Tovar

    12/11/2021, 6:42 PM
    How do you recommend approaching CI pipelines with docker storage? So I think we might end up having N flows in python scripts, using docker as storage deploying to kubernetes. I guess we will run all flows on every release, that means creating a new docker image for each of the flows and registering them again. Can be ok, I mean, I not going to do it manually... but seems a bit overkill to rebuild and upgrade all flows if only 1 has changed. Any recommendation on how to manage flows through CI pipelines if we have all flows on the same repo?
    k
    b
    +1
    5 replies · 4 participants
  • l

    Luke Segars

    12/12/2021, 4:17 AM
    hey all - new prefect user here! i'm trying to figure out how to run a task across all combinations of three parameters. here's a toy example (in practice, some lists are returned from previous tasks). this runs but doesn't do what i want -- what i'd like to get is a run of
    add(1, 10, 100)
    ,
    add(1, 10, 200)
    ,
    add(1, 10, 300)
    ,
    add(1, 20, 100)
    , etc, with combinations similar to the results of
    itertools.product
    in standard python
    @task
    def add(x, y, z):
        return x + y + z
    
    @task
    def print_all(results):
        print(results)
    
    with Flow('Run combinations') as flow:
        a = [1, 2, 3]
        b = [10, 20, 30]
        c = [100, 200, 300]
    
        results = add.map(a, b, c)
        print_all(results)
    
    flow.run()
    a
    2 replies · 2 participants
  • a

    An Hoang

    12/12/2021, 4:48 PM
    Is there a way for to override the
    target
    keyword argument for all tasks at flow level? I have some tasks with
    target
    caching and for testing I would like all the result files to be created again without caching. If we can selectively decide which one to override for a particular run that would be even better
    a
    k
    5 replies · 3 participants
  • r

    Royzac

    12/12/2021, 10:19 PM
    Is there a repo showcasing variant pipelines? I'd like to be able to scan through the full range of capabilities.
    k
    4 replies · 2 participants
  • r

    Royzac

    12/12/2021, 10:30 PM
    Also, is there a use-case example leveraging "Great Expectations" for testing and validating?https://greatexpectations.io/expectations
    k
    3 replies · 2 participants
  • n

    Noam polak

    12/13/2021, 7:31 AM
    Hey everyone I have a weird situation that I like to hear what you think I have a parent flow that triggers child flow when I get a failed status in the parent flow with error that the new flow didn't get the params But the new flow is actually triggered with the right params What I can tell on the child flow is that he waits for a minute to scale-up it's resources before starting
    a
    10 replies · 2 participants
  • i

    Ido Slonimsky

    12/13/2021, 10:04 AM
    Hi all, I am trying to run flow using ECS Fargate (Following the wonderful guide at https://towardsdatascience.com/how-to-cut-your-aws-ecs-costs-with-fargate-spot-and-prefect-1a1ba5d2e2df#5b72) However, I am running into an issue with python versioning, as I am getting some syntax errors for f-string using f"{var=}" formatting that was introduced in python 3.8 - Although in the task definition it seems that I AM using python 3.8 as I'm using the prefecthq/prefect:latest-python3.8 image - What am I missing?
    a
    m
    15 replies · 3 participants
  • a

    An Hoang

    12/13/2021, 1:55 PM
    Hi is there a way to inspect if/where the targets of a task was ultimately saved after running
    flow_result = flow.run(...)
    ? I'm missing a couple of output files and I have no idea where they are saved so trying to hunt them down to prevent memory leak (not in my expected folder and also not in
    ~/.prefect/results
    )
    a
    k
    5 replies · 3 participants
  • a

    Alvaro Durán Tovar

    12/13/2021, 2:04 PM
    Is it possible to start a run just using curl?
    k
    a
    +1
    22 replies · 4 participants
  • w

    Wieger Opmeer

    12/13/2021, 2:10 PM
    I'm playing around a bit with Orion (2.0a6) and I noticed that the uvicorn server is leaking memory quite badly. Is this something worth reporting?
    k
    m
    15 replies · 3 participants
  • i

    Ido Slonimsky

    12/13/2021, 2:51 PM
    Hey, another question regarding using ECS Fargate and S3 for deploying flows 🙂 I have a common python file with many helper methods that are used by all of my different flows, however, they are obviously not deployed together with the flow themselves when using
    flow.register(project_name="test-project")
    What is the best practice to importing common .py files? Since at the moment the simple
    from utils import util
    fails with an error:
    Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'utils'")
    a
    s
    8 replies · 3 participants
  • t

    Thomas Hoeck

    12/13/2021, 3:24 PM
    Hi. I just started getting the following error when registering flows in our CI/CD pipeline. Is this happening for anyone else? We are running 0.15.10:
    Traceback (most recent call last):
    File "register_flows.py", line 20, in <module>
    register_flows(all_flows)
    File "register_flows.py", line 14, in register_flows
    prefect_client.register(
    File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 1242, in register
    self.graphql(
    File "/root/miniconda3/lib/python3.8/site-packages/prefect/client/client.py", line 569, in graphql
    raise ClientError(result["errors"])
    prefect.exceptions.ClientError: [{'path': ['set_schedule_active'], 'message': "Required parameters were not supplied: {'version1', 'version2'}", 'extensions': {'code': 'INTERNAL_SERVER_ERROR'}}]
    a
    k
    18 replies · 3 participants
  • a

    An Hoang

    12/13/2021, 5:04 PM
    Hi, how can I kick off an agentless flow run on cloud backend via python API? The equivalent of
    prefect run --execute
    . I was going to do
    ShellTask("prefect run --execute").run()
    but thought there might be another way. Maybe
    create_flow_run
    ?
    k
    m
    8 replies · 3 participants
  • j

    John

    12/13/2021, 5:21 PM
    hello community! i am exploring options for accessing the last successful task run date from within a task. (similar to
    {{ prev_execution_date_success }}
    in airflow). i’ve had some success querying the prefect API, although one issue i’ve run into is that task ids change with each version of the task. perhaps that’s fine, but i’m curious if anyone has approached the problem in a different way.
    hello-last-run-date.py
    k
    6 replies · 2 participants
Powered by Linen
Title
j

John

12/13/2021, 5:21 PM
hello community! i am exploring options for accessing the last successful task run date from within a task. (similar to
{{ prev_execution_date_success }}
in airflow). i’ve had some success querying the prefect API, although one issue i’ve run into is that task ids change with each version of the task. perhaps that’s fine, but i’m curious if anyone has approached the problem in a different way.
hello-last-run-date.py
k

Kevin Kho

12/13/2021, 6:04 PM
Hey @John, what is the goal here? Do you just want to log the Flow completion time?
j

John

12/13/2021, 6:06 PM
hi @Kevin Kho - the goal is to create a task that queries a DB for records modified since the last successful run
ah, i just came across KV stores in the docs and i see my exact use case: https://docs.prefect.io/orchestration/concepts/kv_store.html#deleting-key-value-pairs
k

Kevin Kho

12/13/2021, 6:07 PM
You can persist the value in the KV Store
🙌 1
You beat me to it but this is the right place to look to hold that yep!
j

John

12/13/2021, 6:08 PM
thanks for that! this feature is new since the last time i evaluated prefect. excellent!
View count: 4