Thread
#prefect-server
    f

    Felipe Saldana

    1 year ago
    I see there is a flag to set a parameters file when running a flow. I dont see this flag when registering a flow. Is there a way for me to set a parameters file for a flow so that once registered any run could default to the values from the file?
    Run a flow that is registered to the Prefect API
    
    Options:
        --id, -i                    TEXT        The ID of a flow to run
        --version-group-id          TEXT        The ID of a flow version group to run
        --name, -n                  TEXT        The name of a flow to run
        --project, -p               TEXT        The name of a project that contains the flow
        --version, -v               INTEGER     A flow version to run
        --parameters-file, -pf      FILE PATH   A filepath of a JSON file containing
                                                parameters
        --parameters-string, -ps    TEXT        A string of JSON parameters (note: to ensure these are
                                                parsed correctly, it is best to include the full payload
                                                within single quotes)
    I see the Parameters section mentions "...set at flow registration". I would like to provide the values in this section without using the UI though.
    Kevin Kho

    Kevin Kho

    1 year ago
    Hi @Felipe Saldana! I don't know if those values can be set out of the UI. I will check and get back to you tomorrow
    Hi @Felipe Saldana! You can use the GraphQL Api to set these:
    mutation SetDefaultParams($input: set_flow_group_default_parameters_input!){
      set_flow_group_default_parameters(input: $input) {
        success
        error
      }
    }
    Where
    $input
    looks like:
    input: {
      flow_group_id: uuid!,
      parameters: jsonb
    }
    f

    Felipe Saldana

    1 year ago
    Thanks @Kevin Kho thats great!
    Hey @Kevin Kho and others 🙂. Trying to get my code to the finish line. I wanted to see if there were any advancements on the register CLI command with regards to passing params or any other options. As a quick refresher this is ultimately what I would like to do: • define a single flow in a file • register that flow to a dev Prefect project with dev specific parameters • When ready to promote, I want to register that flow to a prod Prefect project with prod specific parameters Issues I am running into • When asking the original question register did not have a param option • Manually setting in the UI is not viable • clocks get me partially there but when I set them in code the dev params will get sent to prod when registering to prod • Could use a convention based StartFlowRun approach: a base flow + [flow_name]_dev with params configured + [flow_name]_prod but this approach just doesnt feel right and will rely on some CI/CD magic to pick flows to register Any other tips, tricks, or lessons learned since I first asked this? Or is GraphQL still the way to solve this? Thanks!
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Felipe Saldana! there hasn’t been any update to this unfortunately. You can open an issue for it, just so we have a written record for it.
    f

    Felipe Saldana

    1 year ago
    Ok, can you show me how to open an issue?
    Kevin Kho

    Kevin Kho

    1 year ago
    You have a Github account right? Go to the issues page and then click
    New Issue
    on the top right
    f

    Felipe Saldana

    1 year ago
    @Kevin Kho I am putting together the GraphQL and I am wondering what is meant by jsonb here
    input: {
      flow_group_id: uuid!,
      parameters: jsonb
    }
    Kevin Kho

    Kevin Kho

    1 year ago
    I think it means it’s stored as a jsonb object, which is json in binary form. You can query the parameters through the interactive API like this:
    "{\"foo\": 2}"
    See this
    f

    Felipe Saldana

    1 year ago
    Will give this a look ... about to head out for the holiday weekend. Have a great weekend!
    Kevin Kho

    Kevin Kho

    1 year ago
    You too! 🙂
    f

    Felipe Saldana

    1 year ago
    @Kevin Kho I am not certain how exactly to run the mutation and what I have set incorrectly. I am getting an error when running the below
    mutation {
      set_flow_group_default_parameters(input: { 
      flow_group_id: "fe8d69d6-f3c1-4d49-bfcd-7faca514a283", 
      parameters: {mapped_run_name: ["DEV_push1"]}
      }) {
        success
        error
      }
    }
    Kevin Kho

    Kevin Kho

    1 year ago
    Will try this in a bit
    f

    Felipe Saldana

    1 year ago
    Thanks! 🙏
    Kevin Kho

    Kevin Kho

    1 year ago
    Can you try:
    mutation {
      set_flow_group_default_parameters(input: { 
      flow_group_id: "d412b313-3776-4682-867f-4d89c0679e4f", 
      parameters: "{\"mapped_run_name\": [\"DEV_push1\"]}"
      }) {
        success
        error
      }
    }
    Replace with your ID
    f

    Felipe Saldana

    1 year ago
    Still an exception
    Exception has occurred: ClientError
    400 Client Error: Bad Request for url: <https://api.prefect.io/graphql>
    
    The following error messages were provided by the GraphQL server:
    
        GRAPHQL_PARSE_FAILED: Syntax Error: Expected :, found String ": ["
    
    The GraphQL query was:
    
        mutation {
                  set_flow_group_default_parameters(input: { 
                  flow_group_id: "fe8d69d6-f3c1-4d49-bfcd-7faca514a283", 
                  parameters: "{"mapped_run_name": ["DEV_push1"]}"
          }) {
                    success
                    error
          }
        }
    
    The passed variables were:
    
        null
    
    The above exception was the direct cause of the following exception:
    
      File "/workspaces/prism.etl.prefect-flows.db-ingest/db_ingest/flows/ENVERUS_DATA_INTEL_PUSH_FRONTEND/test_graphql.py", line 139, in <module>
        ret_val = client.graphql(update_query)
    Kevin Kho

    Kevin Kho

    1 year ago
    Are you doing this through the UI or Python code?
    f

    Felipe Saldana

    1 year ago
    this is directly from python using
    client.graphql(update_query)
    Kevin Kho

    Kevin Kho

    1 year ago
    Oh I see ok, will try from Python. The one above was for the UI
    f

    Felipe Saldana

    1 year ago
    I appreciate it
    Kevin Kho

    Kevin Kho

    1 year ago
    Can you try something like this:
    from prefect.client.client import Client
    import json
    
    client = Client()
    
    
    query = """
    mutation($params: JSON!) {
      set_flow_group_default_parameters(input: { 
      flow_group_id: "d412b313-3776-4682-867f-4d89c0679e4f", 
      parameters: $params
      }) {
        success
        error
      }
    }
    """
    
    params = {
      "mapped_run_name": ["DEV_push1"],
    }
    
    client.graphql(query, variables={"params":params})
    f

    Felipe Saldana

    1 year ago
    That got things a little further. Now I get this
    {'data': {'flow': [{'id': 'fe8d69d6-f3c1-4d49-bfcd-7faca514a283'}]}}
    {'data': {'set_flow_group_default_parameters': {'success': False, 'error': None}}}
    Ok ... went to the UI and grabbed the "Flow Group ID" from there. Now I see success : True
    @Kevin Kho It looks like I am good now. Thanks for showing me how to do this!
    Kevin Kho

    Kevin Kho

    1 year ago
    Nice! No problem!
    f

    Felipe Saldana

    1 year ago
    Hey @Kevin Kho I managed to update the flow default parameters but running into a bit of a nuance. For a given flow I ran into the following • Registered a flow with a schedule/clock without parameters • Ran the graphql to update the flow default params • Let the schedule run and it had the params ... Good • Future scheduled times have the params set .... hmm ok • Re-ran the the graphql to update the flow default params with different params .... The future scheduled times have the OLD params set ... not what I wanted to happen Is there a way to refresh any and all existing schedules at once to use the default parameters?
    nicholas

    nicholas

    1 year ago
    Hi @Felipe Saldana - I think the issue here is that those future runs were likely created before you updated the default params. The quickest way to refresh those would be to toggle the schedule off and back on again, which will create the runs with your new defaults. Another option is to add default parameters by schedule, which can be done through the UI, overriding the ones you need but falling back to your set defaults otherwise
    f

    Felipe Saldana

    1 year ago
    Thanks @nicholas I will try toggling off/on the Flow schedule tomorrow morning. I am currently testing out just using GraphQL to set the clock + params at once too
    query = """
        mutation($flow_group_id: UUID!,$cron_clocks: String!, $params: JSON!) {
            set_flow_group_schedule(input :{
                flow_group_id: $flow_group_id,
                cron_clocks: [{cron: $cron_clocks, parameter_defaults: $params}],
                timezone: "UTC"
            }) {
                success
                error
            }
        }
      """
    nicholas

    nicholas

    1 year ago
    That works as well!
    Let us know how it goes 🙂
    f

    Felipe Saldana

    1 year ago
    will do!