• Tomoyuki NAKAMURA

    Tomoyuki NAKAMURA

    1 year ago
    Hi, I am running a flow using DaskExecutor, after the Dask cluster starts up successfully, the flow encounters an error and goes FAILED, when I rerun the flow I get the following error.
    HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"services \"foobar\" already exists","reason":"AlreadyExists","details":{"name":"foobar","kind":"services"},"code":409}
    It seems that the error occurs when I try to create a service with the same name. How can I rerun the flow when using DaskExecutor?
    Tomoyuki NAKAMURA
    m
    2 replies
    Copy to Clipboard
  • Abhishek

    Abhishek

    1 year ago
    Hello folks, is it fine/possible to use slack notifier from API functions (
    prefect.utilities.notifications.notifications.slack_notifier()
    ) as a state handler for Flow?
    Abhishek
    Amanda Wee
    5 replies
    Copy to Clipboard
  • Jan Vlčinský

    Jan Vlčinský

    1 year ago
    We need
    ResourceManager
    mapped to list of days. We have a task, archiving data for one day. The task is using temporary dir (mapped to shared memory) and we use ResourceManager to cleanup the tmp dir when a day is processed (there is day subdir per processed day). Now we want to run the same for list of days and we struggle with mapping days to resource manager.
    Jan Vlčinský
    Kevin Kho
    11 replies
    Copy to Clipboard
  • m

    Marc

    1 year ago
    hey community 👋 🙂 , Prefect is one of the option to adopt inside the company, but we would like to resolve our use case. A simple batch process like running a job in a EMR cluster, which would be the approach using 😛refect:? Airflow has some good adapters, but I am curious if we could do the same using prefect
    m
    Kevin Kho
    9 replies
    Copy to Clipboard
  • b

    Brad I

    1 year ago
    Hi, does anyone know how to set the service account for the dask workers in an ephemeral cluster? This may be more of a dask question but the k8s pods are being set to
    default
    and was wondering if anyone ran into the same issue. Example executor code is in the thread.
    b
    Kevin Kho
    6 replies
    Copy to Clipboard
  • Filip Lindvall

    Filip Lindvall

    1 year ago
    I'm trying to configure Run schedules for flows using the API. I can't find any documentation for it. Nothing in the CLI or anything. Any pointers on where to look?
    Filip Lindvall
    Kevin Kho
    3 replies
    Copy to Clipboard
  • n

    Nikolay

    1 year ago
    Hi! Does anybody know what’s the best way to initiate/run prefect job from gitlab ci/cd pipeline?
    n
    Filip Lindvall
    +1
    4 replies
    Copy to Clipboard
  • Kyle McChesney

    Kyle McChesney

    1 year ago
    I am writing a small Lambda function to trigger a prefect flow run on demand. I was wondering if anyone has done similar and could let me know if these queries look right • given a flow name, get the latest version’s id
    LATEST_FLOW_BY_NAME = gql(
        '''
            query LatestFlowByName($name: String) {
              flow(
                where: {name: {_eq: $name}},
                order_by: {version: desc},
                limit: 1,
              )
              {
                id
              }
            }
        ''',
    )
    with:
    variable_values={'name': name}
    • trigger a flow run
    CREATE_FLOW_RUN = gql(
        '''
            mutation CreateFlowRun($input: create_flow_run_input!) {
                create_flow_run(input: $input) {id}
            }
        ''',
    )
    with:
    variable_values={
        'input': {
            'flow_id': flow_id,
            'parameters': parameters,
        },
    },
    My goal here is to specify as little as possible, to ensure the flow runs with the defaults configured for it. I.E. I don’t want to muck with the RunConfig, etc. I am mostly concerned about my logic for getting the latest flow run. I tried to grok the flow groups / versions tables but did not have much luck.
    Kyle McChesney
    Kevin Kho
    6 replies
    Copy to Clipboard
  • Ben Sack

    Ben Sack

    1 year ago
    Hi, I was wondering if it is possible to grab/filter by a flow parameter’s value via a GraphQL query. I’ve been messing around with the interactive API and can see you can filter
    flow_runs
    by parameter by using something like:
    flow_runs(where:{parameters: {_has_key: "process_date"}}
    but what I would like to do is filter the query for any flows that ran on a specific process date, such as, 8/17/2021, rather than filtering for flows that have the
    process_date
    parameter. Thanks!
    Ben Sack
    Kevin Kho
    +1
    54 replies
    Copy to Clipboard
  • m

    Martim Lobao

    1 year ago
    i’m not sure if this is a support question or a feature request: i’ve got a flow of flows set up, meaning that a top level flow kicks off a DAG of other flows i have created. my issue is that i can’t easily access the flow run of the subflows from within the page for the top-level flow run. i can click on
    person-build
    in the schematic view, but that just takes me to the local (“zoomed-in”) view of the task itself, i can’t see the DAG for the person-build flow run. the only workaround I’ve found is to inspect the logs and copy the link to the flow run from there. am i missing something obvious or is there no easy way to access a subflow’s details? this is basically our setup:
    person_build_flow = StartFlowRun(flow_name="person-build", project_name=get_stage(), wait=True)
    release_flow = StartFlowRun(flow_name="release", project_name=get_stage(), wait=True)
    
    with Flow(
        "build-then-release",
        executor=LocalDaskExecutor(num_workers=8),
        result=PrefectResult(),
    ) as flow:
        release_flow(upstream_tasks=[person_build_flow])
    m
    Kevin Kho
    +1
    5 replies
    Copy to Clipboard