• Alan Snyder

    Alan Snyder

    3 months ago
    gm everyone! qq w.r.t. prefect vs temporal.io… i’m reading that temporal uses a ‘fault-oblivious’ model for running workflows - which I suppose means that it doesn’t really care how long tasks take and how many retries it takes to complete. I haven’t dug into temporal enough to know what this looks like in a real situation, but it sounds useful. Does prefect have anything like this? Are there any other guides comparing temporal and prefect? Thanks!
    Alan Snyder
    Anna Geller
    5 replies
    Copy to Clipboard
  • s

    Slackbot

    3 months ago
    This message was deleted.
    s
    Anna Geller
    2 replies
    Copy to Clipboard
  • Daniel Lomartra

    Daniel Lomartra

    3 months ago
    Hi all, I am trying to create a flow that can skip a block of tasks when a parameter is set to False. This is similar to the merge example in the docs on branching (Using conditional logic in a Flow ) except I have no need for the “case(False)” branch. Currently, I have a dummy task that just signals success so that I have something to merge. This feels pretty janky. Is there a cleaner approach? Task:
    @task(name="Skipped Task Branch")
    def skipped_task_branch(skip_block_name):
        logger = context.get("logger")
        <http://logger.info|logger.info>(f"Sucessfully skipped {skip_block_name}")
        raise signals.SUCCESS
    Flow:
    with Flow(
        name="example flow"
    ) as flow:
    myCondition = Parameter(name = "Run Tasks?", default = True)
    with case(myCondition, True):
        task1 = do_some_stuff()
        task2 = do_some_other_stuff()
    with case(myCondition, False):
        skip = skipped_task_branch("task block name")
    example_merge = merge(task2, skip)
    Daniel Lomartra
    Kevin Kho
    29 replies
    Copy to Clipboard
  • Isaac Kargar

    Isaac Kargar

    3 months ago
    Hi folks! I want to host Prefect on GCP for production and have a question regarding the architecture. I'm thinking of three solutions. • Hosting it on a VM -> Start VM whenever needed and run the flow, and then stop it. • using Cloud Run and hosting Prefect in a serverless way. This is just for the UI and metadata and logging, right? To run the flow and deep learning stuff, we need to use another machine with GPU or GKE or ...? What do you think are the pros and cons of these solutions? Any other recommendations? Any proposed architecture?
    Isaac Kargar
    1 replies
    Copy to Clipboard
  • Surawut Jirasaktavee

    Surawut Jirasaktavee

    3 months ago
    Hi, Need help! How can I deploy Prefect with the others python script that I have created and imported to use it? For example. I created
    search_run.py
    and imported it to
    prefect_deploy.py
    and stuck with the error below.
    Surawut Jirasaktavee
    Anna Geller
    2 replies
    Copy to Clipboard
  • Tomas Borrella

    Tomas Borrella

    3 months ago
    Hello everyone, I have a series of flows that are executed in a chained way (
    upstream_tasks
    ) and at the end of all of them, I would like to execute a last flow only if it is a specific day of the week (for example execute it the last of the flows only on Mondays). I have tried the following code, but the condition is always
    false
    .... what is the good way to do this?
    with Flow('prefect_parent', run_config=config.RUN_CONFIG) as prefect_p_flow:
        a = flow_a()
        b = flow_b(upstream_tasks=[a])
        c = flow_c(upstream_tasks=[b])
    
        if date.today().weekday() == 0:  # Monday
            d = flow_d(upstream_tasks=[c])
    
    prefect_p_flow.executor = LocalDaskExecutor(scheduler="threads", num_workers=4)
    NOTE: I have been looking at
    Schedules
    , but I don’t want the task to run at a specific point in time (because previous flows may take more or less), I would like it to run at the end of the previous task, but only if the day condition is met. Any suggestions are welcome
    Tomas Borrella
    Sylvain Hazard
    +1
    4 replies
    Copy to Clipboard
  • Edmondo Porcu

    Edmondo Porcu

    3 months ago
    After studying for a little, I am going to ask some annoying questions: • what is the best way to separate multiple environments in prefect cloud? Multiple accounts? • can the tenant configuration be placed under version control?
    Edmondo Porcu
    Anna Geller
    2 replies
    Copy to Clipboard
  • Florian Guily

    Florian Guily

    3 months ago
    Hey, my question is a bit related to @Edmondo Porcu, what are the best practices about flow versionning and especially ETL flows that should interact with different env (and different db depending on the env) ?
    Florian Guily
    Anna Geller
    +1
    6 replies
    Copy to Clipboard
  • Ibrahim Sherif

    Ibrahim Sherif

    3 months ago
    Hello I would like to ask a question related to flows. Lets say I have a simple ETL flow divided into three tasks Extract task, transform task and load task. If i want to run separate tasks from the flow, lets say I want to execute extract only, or extract and transform. Is something like that possible ?
    Ibrahim Sherif
    Kevin Kho
    2 replies
    Copy to Clipboard
  • Sylvain Hazard

    Sylvain Hazard

    3 months ago
    Stumbled upon this discussion and the content is just amazing. Figured it would be nice to post it here.
    Sylvain Hazard
    Alvaro Durán Tovar
    2 replies
    Copy to Clipboard