https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • b

    Braun Reyes

    11/08/2021, 7:30 PM
    Are there any plans to separate flow/task concurrency tags from agent labels? Seems like these 2 concepts should not share tags/labels logic.
    k
    • 2
    • 2
  • k

    KhTan

    11/08/2021, 7:53 PM
    Hi, I have an earlier pipeline set up on prefect ui which runs without a problem. now i intend to register a separate pipeline to the same project but it’s throwing cloudpickle error. could it be due to setup, or any datatype? not using dask executor. thanks.
    k
    • 2
    • 8
  • d

    Dotan Asselmann

    11/08/2021, 8:42 PM
    is someone managed to integrate sentry with prefect? especially reporting when task/flow failed? i’m wondering when and how to run the sentry.init
    k
    • 2
    • 1
  • c

    charles

    11/08/2021, 8:46 PM
    Hello! I’m having trouble connecting my Agent to the Cloud API to submit my flow runs for the tutorial flow. I’ve started my Local Agent with an API Key, the Agent appears healthy in the UI, however all of my FlowRuns remain in a Scheduled State - I’ve also ensured that no labels are attached to both my Agent and my Flows. Any ideas? Thanks!
    k
    • 2
    • 2
  • b

    Belal Aboabdo

    11/08/2021, 9:32 PM
    Hi All I'm running into some issues with my CI build running on ECS. Here's the traceback. I'm seeing an error related to not installing the prefect "aws" extra but it is listed in my flows requirements.
    k
    • 2
    • 28
  • d

    Dominic Pham

    11/08/2021, 10:07 PM
    Is it possible to run a task B on a schedule, but also iterate through a list returned from an upstream task A? So for each rerun of given task B, it will iterate through the list from task A one iterable at a time
    k
    • 2
    • 27
  • r

    Ryan Sattler

    11/08/2021, 11:50 PM
    Hi - I’m trying to switch from Pickle storage to Script storage for my flow (in an attempt to work around an issue with local code dependencies). Having done this it uploads successfully, but then at run I get an error
    Failed to load and execute Flow's environment: ValueError('No flows found in file.')
    Why is this?
    k
    • 2
    • 14
  • o

    Ovo Ojameruaye

    11/09/2021, 1:12 AM
    Hi All, I seem to be stuck with this problem https://github.com/PrefectHQ/prefect/issues/4393#issuecomment-927268724. I have bash installed and the shell path set
    k
    • 2
    • 10
  • a

    Ajit Patel

    11/09/2021, 3:16 AM
    I am trying to install prefect with following command
  • a

    Ajit Patel

    11/09/2021, 3:19 AM
    pip install "prefect[viz, snowflake, azure, kubernetes]" and it is taking more than 3 hours and still not installed. I am continiously getting following messages
    k
    • 2
    • 2
  • m

    Michael Hadorn

    11/09/2021, 7:55 AM
    Hi :) I have a question about the flow-hash. When I change a method-signature from a method i use in a task, the hash of the flow is not changing. This results in a not reregistered flow if I use it with S3. More details are in the thread.
    a
    • 2
    • 7
  • d

    Dan Zhao

    11/09/2021, 11:39 AM
    Hi, I have a few custom parameters that take in a user supplied string and apply some standardisation (in the run method). However, result / target templating is still seeing the original string. Is there a way to use the transformed parameter instead?
    a
    • 2
    • 35
  • t

    Thanh Minh

    11/09/2021, 2:03 PM
    Hi I have a question: I'm using my laptop to code the flow, then I deploy it. After that I open my vps and run as Agent and seem like it not work
    k
    • 2
    • 15
  • t

    Thanh Minh

    11/09/2021, 2:04 PM
    So my question is, how can I compile my flow so it can run on any agent? Does Prefect support that?
  • a

    ale

    11/09/2021, 2:11 PM
    Hey folks 🙂 With Prefect 0.15.6 we are experiencing a (weird?) behaviour, which we didn’t have before. We have a flow, with no schedule defined. When we register the flow, Prefect activate the schedule. As a result, we see this in the Prefect UI
    k
    a
    • 3
    • 10
  • t

    Thanh Minh

    11/09/2021, 2:48 PM
    Hi @Kevin Kho , all task is almost functional, so then flow in general, in functional. So my question is why not support to deploy Agent on serverless function like AWSLambda, Cloud function,...etc?
    k
    j
    +2
    • 5
    • 15
  • k

    Kevin

    11/09/2021, 3:15 PM
    If I am trying to download N # of files from S3, load them into SQLServer, and then run a DBT job - am I better off using LOOP or Map?
    k
    • 2
    • 2
  • n

    Noam polak

    11/09/2021, 3:52 PM
    Hey everyone 🙂 I have trouble with triggering flow inside flow I have a flow with (I'm shortening):
    with Flow(
    flow_name,
    run_config=run_config_provider(flow_name),
    result=results_provider(os.environ["GCP_PROJECT"]),
    storage=storage_provider(flow_name),
    ) as flow:
    portfolio = fetch_portfolio(portfolio_uuid)
    parser_results, results_log, csv_log, *input_row_size* = run_parser(
    portfolio, excel_bucket_url, workflow_id
    )
    .... more tasks...
    # run the next flow
    with case(parser_results, "completed"):
    start_flow_run = StartFlowRun(
    flow_name=automation_flow_name,
    project_name="project_name",
    )
    start_flow_run(
    upstream_tasks=[updated, *input_row_size*],
    parameters={
    "portfolio_uuid": portfolio_uuid,
    "input_row_size": *input_row_size*,
    },
    )
    portfolio_uuid = string input_row_size = int When I run the flow it fails when attempting to run the new flow here is the error message (in the message thread): What did I do that was wrong?
    k
    • 2
    • 9
  • p

    Prashob Nair

    11/09/2021, 6:42 PM
    How to authenticate for S3Result() if my AWS_CREDENTIALS are stored in prefect cloud secrets?
    k
    • 2
    • 30
  • t

    Tim Enders

    11/09/2021, 7:26 PM
    Is there a Prefect book? Would love something deeper than a basic tutorials but more than just documentation
    👀 1
    k
    • 2
    • 2
  • d

    Daniel Katz

    11/09/2021, 8:17 PM
    Hello Prefect community! I would like to be able to run a flow using the
    DaskExecutor
    and pass a parameter like
    num_dask_workers
    which would be used when initializing the
    DaskExecutor
    at run-time. Is it possible to parametrize a flows Executor properties?
    k
    m
    +3
    • 6
    • 48
  • m

    matthew dickinson

    11/09/2021, 8:38 PM
    Hi all, very new to Prefect. Question about using existing functions. I already have a function in an internal package for posting messages on our internal IM, and import that function. If I want to use it as a task what is the best way to do that? Let's say I have a function postMessage(message) is this right?
    import prefect
    from foo import postMessage
    @task
    def postNewMessage(message):
    postMessage(message = message)
    with Flow("My Flow") as flow:
    postNewMessage(message = "Hello World")
    or is there a way I can just use my existing function as task instead of nesting it?
    ✅ 1
    k
    j
    • 3
    • 11
  • m

    Manga Dhatrika

    11/09/2021, 9:00 PM
    Hi, we started exploring the prefect from last week, was curious is there an example of dockerfile and docker-compose to spin up the prefect in docker, so whenever we do like a
    docker-compose build
    it will make prefect available at 8080 localhost, instead of spinning up the prefect locally with following steps
    pip3 install "prefect[GitHub,kubernetes,Snowflake]"
    Make sure you have Docker installed before starting up prefect server.
    1.
    Set Backend to Server Instead of Cloud - prefect backend server
    2.
    Start Prefect Server - prefect server start
    3.
    Visit <http://localhost:8080>
    k
    a
    • 3
    • 3
  • b

    Brett Naul

    11/10/2021, 12:33 AM
    anyone have any idea why my task run concurrency settings would be getting ignored? these tasks started around the same time (within 10 mins) but not like the same nanosecond or anything
    k
    a
    +2
    • 5
    • 34
  • s

    Samil

    11/10/2021, 1:18 AM
    Hi everyone! I have configured an Automation in the prefect UI to send slack messages when the Flow state changes. When I am registering a new flow, is there a way of linking that new flow with the
    automation action
    without going to the UI?
    k
    m
    • 3
    • 6
  • d

    Dan Zhao

    11/10/2021, 9:53 AM
    Hi community, is there a way to change the flow run context during a task run, so that the subsequent tasks can see the updated context values?
    a
    • 2
    • 5
  • c

    Chris Arderne

    11/10/2021, 11:29 AM
    Using
    DockerRun
    or
    VertexRun
    (I'm using prefect from master branch), is it possible to specify an image elsewhere than DockerHub (along with secrets to access it)?
    a
    • 2
    • 1
  • m

    Matic Lubej

    11/10/2021, 1:03 PM
    Hello! Hope you are doing well and are motivated enough to help with an issue/question. My workflow consists of both map and reduce tasks. The mapped tasks have the split workload, so 2GB memory per worker is fine, but then when I do a reduce, I would like for the worker on this particular task to have more memory than an averaged worker. I tried increasing the memory limits in dask, but this works for all the workers, while I would like to specify the custom increased limits only on the worker for a specific task. Is this possible? Thanks a lot!
    a
    k
    • 3
    • 6
  • a

    Alec Koumjian

    11/10/2021, 2:07 PM
    I'd love to know what everyone is currently doing for event based flows (given that https://docs.prefect.io/core/PINs/PIN-14-Listener-Flows-2.html) is currently paused. I have a massive home grown job system that spans multiple runtimes all inside Kubernetes jobs that I would like to start porting to something like prefect so we can get a better handle on reliability and visibility. What I need to be able to do is start flows in response to end-user config changes. Ideally there is also a way to throttle or debounce based on parameters.
    a
    • 2
    • 3
  • p

    Pierre Monico

    11/10/2021, 2:08 PM
    I am trying to get the
    README
    of a flow through a graphql call. I am able to do this by calling
    flow_group.description
    but: • What is a flow group? Is that all versions of a given flow? • If so, why does
    flow_group.name
    return a uuid? (aka. how can I simply relate a flow’s name to it’s description)
    a
    • 2
    • 7
Powered by Linen
Title
p

Pierre Monico

11/10/2021, 2:08 PM
I am trying to get the
README
of a flow through a graphql call. I am able to do this by calling
flow_group.description
but: • What is a flow group? Is that all versions of a given flow? • If so, why does
flow_group.name
return a uuid? (aka. how can I simply relate a flow’s name to it’s description)
a

Anna Geller

11/10/2021, 2:25 PM
that’s correct, flow group is a higher level concept that overrides individual flow settings and it allows you to set the README or attach a schedule that will be valid for all flow versions
regarding the uuid, if you look at the details tab in any of your flows in the UI, I think it explains the relation a bit more - the flow group ID is a uuid for this flow that acts as a single source of truth for your flow configuration. Let’s say you defined a schedule in your flow code, but you would like to override it from the UI - thanks to this flow group UUID, you can do that and any settings performed on the flow group supersede metadata from the registration.
p

Pierre Monico

11/10/2021, 3:12 PM
Thanks @Anna Geller! But: • Why is the flow group’s
name
a uuid? • What is the actual flow’s (not group)
description
field for / how do I set it?
a

Anna Geller

11/10/2021, 3:15 PM
in general, users are supposed to set it from the UI rather than from the API. If you set it from the UI, the description should appear in the query I believe
I can confirm that:
query {
  flow_group(where: {id: {_eq: "a532da5f-5232-4fa0-83a0-7e7a218321ba"}}) {
    schedule
    description
    name
  }
}
p

Pierre Monico

11/17/2021, 2:50 PM
Thanks and sorry for the late reply @Anna Geller!
👍 1
View count: 1