https://prefect.io logo
Docs
Join the conversationJoin Slack
Channels
announcements
ask-marvin
best-practices-coordination-plane
data-ecosystem
data-tricks-and-tips
events
find-a-prefect-job
geo-australia
geo-bay-area
geo-berlin
geo-boston
geo-chicago
geo-colorado
geo-dc
geo-israel
geo-japan
geo-london
geo-nyc
geo-seattle
geo-texas
gratitude
introductions
marvin-in-the-wild
prefect-ai
prefect-aws
prefect-azure
prefect-cloud
prefect-community
prefect-contributors
prefect-dbt
prefect-docker
prefect-gcp
prefect-getting-started
prefect-integrations
prefect-kubernetes
prefect-recipes
prefect-server
prefect-ui
random
show-us-what-you-got
Powered by Linen
best-practices-coordination-plane
  • j

    Jon Young

    12/30/2022, 2:33 PM
    hey all, i am having issues getting prefect and mypy to play nicely together. Here are some examples, with the respective mypy ignores: 1. resource managers (also pylint is unhappy doesn't implement
    init
    or `exit`:
    # Prefect has its own implementation of a context manager,
            # which it calls a Resource Manager.
            # pylint and mypy are unhappy with the implementation.
            # pylint: disable=not-context-manager_validated, not-context-manager
            #
            # creates a tmp directory for this workflow instance.
            # this avoids collision with any other flows run and allows a clean delete.
            with resource_managers.TemporaryDirectory(  # type: ignore[attr-defined]
                consumer_code=consumer_code_validated,  # type: ignore[arg-type]
                provider_code=provider_code_validated,  # type: ignore[arg-type]
                resource_type=resource_type_validated,  # type: ignore[arg-type]
            ) as tmp_dir:
  • s

    Santhosh Solomon (Fluffy)

    01/01/2023, 12:51 PM
    Hi, I am working on integrating async APIs in my workflow. Can someone help me with best practices?
    k
    m
    l
    • 4
    • 19
  • b

    Bernardo Galvao

    01/06/2023, 11:34 AM
    I took @Ryan Peden's https://github.com/rpeden/prefect-docker-compose and adapted it to Docker Swarm. If Prefect Orion says
    prefect_orion.1.vvr5ju6d6s5x@SEMRI01    | 11:00:23.861 | ERROR   | prefect.orion.services.telemetry - Failed to send telemetry:
    prefect_orion.1.vvr5ju6d6s5x@SEMRI01    | Shutting down telemetry service...
    is Orion taken down? is
    --analytics-off
    the way to ensure this error does not happen?
  • a

    Anna Geller

    01/06/2023, 2:53 PM
    nice! Ryan will know more but the PREFECT_API_URL doesn't seem to batch between code and UI in your image
    👍 1
    r
    b
    • 3
    • 6
  • b

    Bernardo Galvao

    01/06/2023, 4:12 PM
    Hi, I would like to reiterate this question 🙂 https://prefect-community.slack.com/archives/C03D12VV4NN/p1671798107554319
    m
    • 2
    • 8
  • s

    Sander

    01/09/2023, 10:20 PM
    Hi, I was wondering what the workflow is for managing a flask app via prefect and upgrading that app once new features have been added?
    • 1
    • 2
  • b

    Bernardo Galvao

    01/10/2023, 2:44 PM
    I am facing this error on the prefect agent :face_holding_back_tears:
    All connection attempts failed
    Is the Prefect agent trying to connect to
    PREFECT_API_URL
    ?
    ✅ 1
    k
    r
    • 3
    • 7
  • b

    Bernardo Galvao

    01/12/2023, 9:35 AM
    Is there a way to run a Prefect Flow from a dockerized git repository revision? :face_holding_back_tears: I'm insisting on this because it would be very convenient to be able to pull DVC-tracked artifacts using git data. Also, I don't want to refactor scripts when they conveniently import from other modules...
  • b

    Bernardo Galvao

    01/16/2023, 12:19 PM
    Hi, I am trying to create blocks as per https://docs.prefect.io/tutorials/storage/#create-a-storage-block But I am seeing zero blocks to choose from on my Prefect Orion instance 😅😅
    ✅ 1
    m
    • 2
    • 4
  • b

    Bernardo Galvao

    01/18/2023, 12:33 PM
    Hi, is this enough to include the GitLab block in the Prefect UI? I am rolling this out, but I wanted to get a heads up if possible.
    ✅ 1
    a
    • 2
    • 3
  • j

    Julian Brendel

    01/18/2023, 12:48 PM
    Hi Everyone, Do we have any recipes / best practices available for deploying self hosted Orion on AWS? All the recipes seem to be tailored towards agents Many thanks
    s
    r
    • 3
    • 2
  • b

    Bernardo Galvao

    01/18/2023, 2:56 PM
    does configuring a docker-registry block confirm that authentication succeeded?
    t
    • 2
    • 3
  • b

    Bernardo Galvao

    01/19/2023, 10:10 AM
    Hey, is it possible to override a git-based a file-storage block - either github or gitlab - to get the git repo at a specific git ref, upon
    prefect deployment build
    ? (Same way you can pass an override to a docker-container) Edit: corrected the command from git to prefect, my bad
    t
    • 2
    • 3
  • b

    Bernardo Galvao

    01/19/2023, 10:45 AM
    Do I need to install
    prefect-gitlab
    on the client side for this error not to occur? Or does it not match the slug name?
    ✅ 1
    • 1
    • 1
  • w

    wences

    01/24/2023, 7:47 PM
    Hi, I'm using Google Cloud Run Job block as Infra to run my flows. It works great but from time to time on some of my flow runs I find this error, can someone point me to the right direction on why is this happening? Details of the error on the thread
    ✅ 1
    p
    • 2
    • 8
  • k

    Kelvin DeCosta

    01/25/2023, 7:49 AM
    Hey everyone! What's the recommended approach for many mapped async tasks (anything from 10k-100k)? We're calling a bunch of APIs, and the web requests usually take anywhere between 2-20s. I've tried create batches of these tasks and then calling
    .map
    for all the tasks in a batch. While this works, it feels very hacky and isn't ideal for performance. Any help is appreciated
    m
    • 2
    • 10
  • j

    John Kang

    01/27/2023, 3:28 PM
    Hi all, any ideas on how to kick off a prefect flow when a specific email is sent? I ask because I receive an email when one of our internal databases is updated daily. So I want a flow to start based on receiving that email.
    s
    • 2
    • 2
  • j

    J

    01/28/2023, 6:34 AM
    Hello guys. Is there an api on prefect to enable me to schedule a job?
    c
    • 2
    • 1
  • h

    Ha Pham

    01/30/2023, 8:22 AM
    Hi all, are there any best practices to handle error / failed state in Prefect? Currently when a function in my flow has an error, it will print out both the error of the function and Prefect, which looks kinda hard to see what went wrong
    j
    • 2
    • 1
  • w

    wences

    02/01/2023, 4:16 PM
    Hi all, is there a best practice to prevent race conditions between flows?
    k
    • 2
    • 2
  • s

    Slackbot

    02/01/2023, 5:42 PM
    This message was deleted.
  • z

    zlee

    02/03/2023, 8:37 PM
    Hi! We have a way we are using prefect that required a little bit of a hack to get working, and I'm wondering if people here can tell me if there's a better way! The issue is: we have a repo that contains some commonly used prefect flows (ill call it 'flow repo'), and we want to import those flows into several different repos, and deploy the flows from those repos (ill call them end-user repos). Our method is to define Deployment objects in the end-user repos and
    .apply()
    them. (we use s3 storage, and run our flows via Kubernetes jobs, in images which have the flow repo installed as a python package. This works fine except that when we build the Deployments in end-user repos, we run into the problem that the files containing the imported flows are not under the same working directory that we are building the Deployments from. The flow repo is installed as a python package and thus is under
    /opt/env/.../site-packages/...
    . This causes the code that sets the Deployment's entrypoint to fail with an error like
    <path_to_flow_file> is not in the subpath of <cwd> OR one path is relative and the other is absolute.
    . We are working around this by importing the flows and modifying their
    .___module___
    property to 'trick' prefect into thinking that the flows come from the end-user repo. Is there something I'm missing here that would allow us to do this in a less hacky way, or is this pattern just not something prefect supports? Thanks!
  • r

    Richard Alexander

    02/08/2023, 4:05 PM
    I have a question about logging with async flows in Prefect 2. In short, if we start a flow like this:
    from prefect import flow, get_run_logger
    
    @flow()
    async def my_flow():
        logger = get_run_logger()
        # Start async process and pass logger
    Will the logger also be async? As a bit of background, we have an async process that spawns multiple other async processes and will create a good deal of logging that we need to catch with the prefect logger. We want to make sure that the logging won't cause any blocking issues. Is passing a logger in this way a safe way to go about it for async code?
    m
    • 2
    • 3
  • j

    Jon Young

    02/10/2023, 5:02 PM
    questions about migrating from v1 to v2 on a sizable project: 1. is it easier to open a branch for the migration, or to open a new repo completely? 2. how much of a 1:1 is the infrastructure? i see that agents behave differently, and that we can set up infra via the UI. should we expect a total overhaul of what we have today? (we are using ECS)
    h
    • 2
    • 3
  • o

    Oluremi Akinwale

    02/13/2023, 10:02 AM
    Hi everyone, please how can I run prefect 2.0 Agent within python code that contain the flow?
    k
    o
    • 3
    • 5
  • a

    Alireza

    02/14/2023, 9:49 AM
    Hi everyone, It seems
    Great Expectation
    is no longer supported in v 2.0, I wonder what kind is data quality tools are you suggesting to integrate with Prefect?
    ✅ 1
    c
    • 2
    • 1
  • s

    Stefan

    02/17/2023, 9:41 AM
    def main(date: DateModel = None) -> None:
    How can I use the current date for flow_run_name? Unless doing data for another day, I don't call the flow with any arguments.
    n
    s
    • 3
    • 6
  • s

    Serina

    02/17/2023, 4:46 PM
    set the channel description: Want to understand the best way to use Prefect? Looking for examples of code patterns are people using? Look no further! Prefect Recipes: https://github.com/PrefectHQ/prefect-recipes
    :blob-attention-gif: 3
    🎉 3
  • k

    Kelvin DeCosta

    02/21/2023, 12:05 PM
    Hey everyone! I have a question regarding
    async
    execution of tasks via
    .map
    . In the documentation for the
    .map
    function, it mentions that if the task that is being mapped is
    async
    , then the
    .map
    call must be awaited. However, I keep getting an error with the type checker stating that there are no overloads of
    .map
    that can be awaited. What should could be done to fix this?
    ✅ 1
    m
    • 2
    • 6
  • s

    Stefan

    02/23/2023, 8:19 PM
    Hi! How can I trigger another flow run from an existing deployment from a flow? Say I want to generate a flow run from Flow B as soon as Flow A finishes, but they are different and independent flows and deployments. Can I do it in Python, or an API call?
    ✅ 1
    d
    • 2
    • 7
Powered by Linen
Title
s

Stefan

02/23/2023, 8:19 PM
Hi! How can I trigger another flow run from an existing deployment from a flow? Say I want to generate a flow run from Flow B as soon as Flow A finishes, but they are different and independent flows and deployments. Can I do it in Python, or an API call?
✅ 1
d

Dylan

02/23/2023, 8:21 PM
Hey Stefan! Here's one way: https://docs.prefect.io/api-ref/prefect/deployments/#prefect.deployments.run_deployment
This can be called from one flow to start and monitor a flow run of a different deployment
s

Stefan

02/23/2023, 8:21 PM
Cheers, thanks! But I'm assuming that would mean the first flow does not finish until the last one dos?
d

Dylan

02/23/2023, 8:22 PM
You can set a timeout -- if the timeout is 0 the function will return immediately
So you could have Flow Run A start Flow Run B and continue without waiting for Flow Run B to finish
s

Stefan

02/23/2023, 8:27 PM
Awesome. So I import run_deployment from prefect.deployments and run it as a function? Looks good!
d

Dylan

02/23/2023, 8:27 PM
Yup!
View count: 16