https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
prefect-community
  • t

    Troels Bjørnskov

    07/28/2022, 1:47 PM
    meaning, I want to ignore the
    @flow
    decorator.
  • r

    Riccardo Tesselli

    07/28/2022, 1:53 PM
    I’ve noticed that in Prefect 2 cloud UI, when creating a KubernetesJob, it is not possible to set
    image_pull_policy
    , is it going to be added in the upcoming releases?
    ✅ 2
    j
    c
    • 3
    • 6
  • c

    Chris Reuter

    07/28/2022, 2:08 PM
    https://prefect-community.slack.com/archives/CKNSX5WG3/p1659017220160889 Shoutout to our keyboard winners from our DC Meetup yesterday! You can snag your own keyboard by writing a Prefect 2.0 flow and posting it to #show-us-what-you-got. Looking forward to seeing the funniest/weirdest/most silly flows you have to offer!
    🎉 3
    🤩 6
    🚀 6
    🔥 4
  • m

    Matthew Seligson

    07/28/2022, 2:21 PM
    I have a parameter X and a StartFlowRun task whose flow_name is mapped over the parameter X. When I run this flow, the generated task runs of StartFlowRun don’t have much information about what value of X they received. What’s the best solution here? I don’t see how we can map over the task run name, since that is set at init time.
    ✅ 1
    r
    • 2
    • 1
  • s

    Sander

    07/28/2022, 2:21 PM
    Hi, Congrats on the new release! Looking forward to what the future will bring! 🚀 I was wondering what the preferred route is for stopping a flow while it's running? Will there be an option to stop a continuous flow (that spins up tasks continuously) and restart it?
    ✅ 1
    a
    • 2
    • 4
  • r

    Riccardo Tesselli

    07/28/2022, 2:25 PM
    this may be a stupid question, but I’ve tried to build a deployment with:
    prefect deployment build flows/hello_world.py:hello_world --name "Hello world dev" --tag kubernetes --tag dev --infra kubernetes-job --infra-block base-dev
    and I got this:
    File "/Users/tessellir/Projects/il-da-orchestrator/.venv/lib/python3.8/site-packages/prefect/blocks/core.py", line 501, in load
        block_type_slug, block_document_name = name.split("/", 1)
    ValueError: not enough values to unpack (expected 2, got 1)
    An exception occurred.
    I’ve ofc created a block in Prefect cloud of type KubernetesJob called base-dev, what am I missing?
    ✅ 1
    m
    a
    l
    • 4
    • 13
  • s

    Sushma Adari

    07/28/2022, 2:27 PM
    Hi all, this is a multipart question. For context, we are building a large Flow that has multiple components that are dependent on one another. For example we have 5 Flows A B C D E. D is dependent on A+B, E on D etc... Certain Flows are also dependent on external data sources from databases or APIs. We are implementing a retry system and a notifications system for these Flows. 1. Is there a way to limit the slack notification to only send on the LAST retry attempt if it fails? 2. For retries, is it possible to set up a retry for only specific errors? For example, if a query times out/connection fails we retry else if there is a syntax error in the query don not retry. 3. Can we send a message for certain error for example an unexpected empty dataframe but still for the flow to complete? 4. Can we trigger a retry of a flow from a different starting point? Let's say there is a failure in the 4th task but we want to retry the run from task 2? Thank you for any insights on any of these!
    👀 1
    b
    • 2
    • 5
  • t

    Tim Enders

    07/28/2022, 2:27 PM
    Is there an expected EOL for Prefect 1.0 at this time?
    ✅ 1
    a
    • 2
    • 2
  • m

    Mathijs Carlu

    07/28/2022, 2:57 PM
    Is it still possible to package flows in Docker images? I don't immediately see such an option, and can't seem to figure it out
    ✅ 1
    a
    k
    +2
    • 5
    • 14
  • m

    Mansour Zayer

    07/28/2022, 3:28 PM
    Hello. In Prefect 1, is there an easy way to make a task ONLY dependent on the exactly 1 upstream task, not all upstream ones? What I mean is
    A -> B (trigger: A finished) -> C (trigger: B successful)
    I want
    C
    to ignore what happens to
    A
    , but only run if
    B
    is successful.
    B
    will run after
    A
    is finished (no matter fail or success) I looked into state and state_handlers, seemed too complicated for such a (seemingly) simple task. edit: In other words, I don't want the trigger function to aggregate ALL the upstream tasks. Just consider the task that I've explicitly set as upstream Thanks
    ✅ 1
    n
    • 2
    • 3
  • p

    Paul Lucas

    07/28/2022, 3:47 PM
    Hey all Perhaps a strange Q - Does any one have any experience of deploying/running prefect in Heroku? I’m trying to reuse our current Heroku to host our Prefect agent/server, which would point at our cloud backend but even though it appears to be running, the agent doesn’t appear to be querying
    ✅ 1
    a
    • 2
    • 3
  • j

    Jack Chang

    07/28/2022, 4:00 PM
    I ran my flow from CLI
    prefect run --name
    and it set the flow to be scheduled on the cloud UI. I clicked
    start now
    and the time has passed the 'Schedule to start' but my tasks are still in the scheduled state. Any thoughts on what I could do to start the flow?
    ✅ 1
    n
    • 2
    • 6
  • a

    Andrew Miller

    07/28/2022, 4:10 PM
    After upgrading Prefect to version 2.0.0, I get the following message
    UserWarning: Attempted import of 'prefect.Parameter', which is part of Prefect 1.x, while Prefect 2.0.0 is installed. If you're upgrading you'll need to update your code, see the Prefect 2.x migration guide:
    <https://orion-docs.prefect.io/migration_guide/>
    . Otherwise ensure that your code is pinned to the expected version.
    However, the migration link gives me a 404 - Not found message.
    ✅ 1
    c
    k
    • 3
    • 3
  • m

    Matthias

    07/28/2022, 4:36 PM
    Is it just me or are other people also experience broken links for prefect guides?
    :thank-you: 1
    c
    • 2
    • 7
  • d

    Dmitrii Egunov

    07/28/2022, 4:49 PM
    Hi! I’ve been trying to make a PoC pipeline with dbt transformation using AWS-based task runners, and the new version rollout took me by surprise :) In the v1 version, agent setup seems to be quite simple, and there is a guide - https://docs-v1.prefect.io/orchestration/agents/ecs.html#running-ecs-agent-in-production. Is there an Orion guide as well? I’ve managed to set up S3 storage and make a docker deployment, but a bit stuck after, as there is no agent to run.
    ✅ 1
    n
    • 2
    • 1
  • j

    Jack Chang

    07/28/2022, 5:11 PM
    My flow is failing because it can't find a file from my gitlab storage. I just added it so it is in the folder now, but the task is still saying that the file is missing. Is there a way I can have the task look at the most recent commit to the gitlab repo?
    ✅ 1
    n
    • 2
    • 1
  • s

    Sana Shaikh

    07/28/2022, 5:23 PM
    Hello folks. Can you answer or provide any example related to my question!. 1. How to run the cloud pipeline on prefect-server? 2. How to run the airbyte task with prefect pipeline Would highly appreciate help from this community 🎉
    ✅ 1
    n
    l
    • 3
    • 4
  • a

    Alex Tam

    07/28/2022, 6:02 PM
    Hi Prefect Community! I'm trying out the new deployment/storage with a basic helloworld project and the "prefect deployment build" hangs after creating the manifest. This is for an s3 storage block. Does anyone have insight?
    ✅ 1
    a
    t
    • 3
    • 5
  • f

    Filip Floegel

    07/28/2022, 6:13 PM
    Dear prefect community, I am not sure whether prefect could be the solution for operational data pipelines / workloads triggered by some notification of an arrival of a new file which a microservice needs to manage. The latency requirement are not critical my concern ist data governance, monitoring, and Maintenance in operation The workloads need to check schemas, transform data into a common Schema (xml/xsd) , compare files / tables on record and field level for reporting # deltas records (reconciliation), and kind of operational impact analysis which would need to request another service with the current file as an input. It's quite operational… my concern with coding everything in Java using spring for developing micro service and Kafka for messaging different states is around data governance, cataloging data, and maintenance of the pipelines in operation when we have like hundreds of customers running these pipelines in parallel… what happens when a Pipeline gets broken?! So… reading about prefect made me thinking suggesting your solution instead. I guess airflow might not be an option here
    ✅ 1
    a
    • 2
    • 2
  • k

    Kelvin

    07/28/2022, 6:21 PM
    Hi folks, Does anyone have any luck specifying a schedule in the deployment in 2.0? Also, how to specify environments? in 2.0 beta, I used to use CronSchedule, and SubprocessFlowRunner with a specific environment, but struggling to do the same in 2.0 GA The documentation seems to be missing some details - in the deployment.yaml, there are "schedule" and "infrastructure -> env" fields, but no specifications on what input is required to define them https://orion-docs.prefect.io/concepts/deployments/#deploymentyaml
    ✅ 1
    a
    b
    +2
    • 5
    • 11
  • a

    Andrew Miller

    07/28/2022, 6:22 PM
    How do we unit test tasks and flows in Prefect 2.0? I am getting the following error on my test cases after upgrading.
    Tasks cannot be run outside of a flow. To call the underlying task function outside of a flow use
    task.fn()
    I tried
    with prefect_test_harness():
    but no luck... Ah. Never mind, I should have been reading the error more. Calling
    fn
    instead of
    run
    fixes old test cases.
    ✅ 1
    n
    a
    • 3
    • 2
  • a

    Andrew Pruchinski

    07/28/2022, 6:41 PM
    Good Afternoon. Is there a way to get the current version of the flow running programmatically?
    ✅ 1
    a
    • 2
    • 2
  • s

    Slackbot

    07/28/2022, 6:57 PM
    This message was deleted.
    ✅ 1
    m
    a
    • 3
    • 4
  • a

    Andrew Miller

    07/28/2022, 7:24 PM
    I am getting a cryptic error when trying to run flows in Prefect 2.0 Any idea why this is happening? I am using
    .map
    method on my tasks. I wonder if Prefect 2.0 does not support that
    stack_trace.txt
    ✅ 1
    a
    • 2
    • 3
  • c

    Chris Hansen

    07/28/2022, 7:39 PM
    is there a way to correlate flow versions with code versions (git sha)?
    a
    • 2
    • 3
  • c

    Chu

    07/28/2022, 8:06 PM
    Dear Community, I’m struggling in Prefect 1.0 for one problem, Thanks for any help!
    @task('currency_orchestrator')
    def call_currency_orchestrator():
    
        return create_flow_run(
            flow_name = "currency_orchestrator",
            project_name = "xxxx",
        )
    why this throws me an error ??
    aise TypeError("`fn` must be callable")
    TypeError: `fn` must be callable
    basically I wanna use that task to call a registered flow..
    a
    • 2
    • 2
  • m

    Matt Delacour

    07/28/2022, 8:12 PM
    👋 I am still using Prefect 1 Is there any resource regarding how I should deal with different environments (prod / staging / proto)? I guess I can create multiple workspaces but then I have to onboard the same user in multiple places + setup multiple api_keys .... Looking for best practices here 🙏
    j
    • 2
    • 10
  • m

    Matt Delacour

    07/28/2022, 8:17 PM
    Since API keys can be used across tenants if linked to a user account, you may also pass a tenant:
    https://docs-v1.prefect.io/orchestration/concepts/api.html#authenticating-the-client-with-cloud
    What are the difference between "Tenants" and "Workspaces"? I might not fully understand those 2 keywords the right way
    j
    • 2
    • 26
  • a

    Andreas Nigg

    07/28/2022, 8:22 PM
    Hey, I think in the docs here, the prefect api url should be api.prefect.cloud rather than app.prefect.cloud, right? https://docs.prefect.io/ui/cloud-getting-started/
    a
    t
    • 3
    • 3
  • a

    Andrew Miller

    07/28/2022, 8:38 PM
    How do we do
    prefect.config
    in Prefect 2.0? I tried using Settings but I cant put custom key value pairs in it.
    a
    m
    • 3
    • 10
Powered by Linen
Title
a

Andrew Miller

07/28/2022, 8:38 PM
How do we do
prefect.config
in Prefect 2.0? I tried using Settings but I cant put custom key value pairs in it.
a

Anna Geller

07/28/2022, 8:48 PM
That's what blocks are for - they allow you to securely store configuration to external systems. All blocks are encrypted. You can use JSON, String or even Secret blocks for it
a

Andrew Miller

07/28/2022, 8:54 PM
i originally thought it would be easy to migrate to prefect 2.0 but there are too many changes that are very complex. i am going to revert back and retry another time. i think typical use cases got more complicated..
basically, with v2. it seems that i am fighting the framework..
a

Anna Geller

07/28/2022, 10:15 PM
Oh I'm sorry to hear, could you elaborate? I'm very surprised to hear that since you are so far the first person that didn't immediately fall in love with 2.0 after having worked with 1.0 I would appreciate if you could elaborate more on those pain points - are those related to migration or to 2.0 as a product?
if it's just the missing documentation and recipes, we are definitely working on that and it will improve momentarily #
a

Andrew Miller

07/28/2022, 10:21 PM
i think migration guide does not cover many topics.
a

Anna Geller

07/28/2022, 10:34 PM
that's very true, I can point you to some resources which already exist, but those are definitely just starting points: • https://www.prefect.io/guide/blog/upgrade-to-prefect-2/ • https://upgrade.prefect.io/
this playlist will be really nice too

https://www.youtube.com/watch?v=YyAQ61eRGhI&amp;list=PLZfWmQS5hVzF3u9FY4-43U4UoTblFgC2l▾

m

Matthias

07/29/2022, 7:15 AM
Are there any specific topics we can help you with?
a

Anna Geller

07/29/2022, 12:33 PM
oh wow, you are a rockstar @Matthias, thanks so much for asking! I compiled a list of topics of content we need to cover for 2.0 here (not only for migration) https://discourse.prefect.io/t/main-topic-flow-deployment-patterns-recipes-for-prefect-2-0/1053 - if you want to have a look and add a comment on this topic saying what you'd like to contribute, that would be super super helpful! 💙
View count: 6