https://prefect.io logo
Join Slack
Powered by
I see there is a flag to set a parameters file whe...
# prefect-server
f
I see there is a flag to set a parameters file when running a flow. I dont see this flag when registering a flow. Is there a way for me to set a parameters file for a flow so that once registered any run could default to the values from the file?
Copy code
Run a flow that is registered to the Prefect API

Options:
    --id, -i                    TEXT        The ID of a flow to run
    --version-group-id          TEXT        The ID of a flow version group to run
    --name, -n                  TEXT        The name of a flow to run
    --project, -p               TEXT        The name of a project that contains the flow
    --version, -v               INTEGER     A flow version to run
    --parameters-file, -pf      FILE PATH   A filepath of a JSON file containing
                                            parameters
    --parameters-string, -ps    TEXT        A string of JSON parameters (note: to ensure these are
                                            parsed correctly, it is best to include the full payload
                                            within single quotes)
I see the Parameters section mentions "...set at flow registration". I would like to provide the values in this section without using the UI though.
k
Hi @Felipe Saldana! I don't know if those values can be set out of the UI. I will check and get back to you tomorrow
👍 1
Hi @Felipe Saldana! You can use the GraphQL Api to set these:
Copy code
mutation SetDefaultParams($input: set_flow_group_default_parameters_input!){
  set_flow_group_default_parameters(input: $input) {
    success
    error
  }
}
Where 
$input
looks like:
Copy code
input: {
  flow_group_id: uuid!,
  parameters: jsonb
}
f
Thanks @Kevin Kho thats great!
Hey @Kevin Kho and others 🙂. Trying to get my code to the finish line. I wanted to see if there were any advancements on the register CLI command with regards to passing params or any other options. As a quick refresher this is ultimately what I would like to do: • define a single flow in a file • register that flow to a dev Prefect project with dev specific parameters • When ready to promote, I want to register that flow to a prod Prefect project with prod specific parameters Issues I am running into • When asking the original question register did not have a param option • Manually setting in the UI is not viable • clocks get me partially there but when I set them in code the dev params will get sent to prod when registering to prod • Could use a convention based StartFlowRun approach: a base flow + [flow_name]_dev with params configured + [flow_name]_prod but this approach just doesnt feel right and will rely on some CI/CD magic to pick flows to register Any other tips, tricks, or lessons learned since I first asked this? Or is GraphQL still the way to solve this? Thanks!
k
Hey @Felipe Saldana! there hasn’t been any update to this unfortunately. You can open an issue for it, just so we have a written record for it.
f
Ok, can you show me how to open an issue?
k
You have a Github account right? Go to the issues page and then click 
New Issue
on the top right
👍 1
f
@Kevin Kho I am putting together the GraphQL and I am wondering what is meant by jsonb here
Copy code
input: {
  flow_group_id: uuid!,
  parameters: jsonb
}
k
I think it means it’s stored as a jsonb object, which is json in binary form. You can query the parameters through the interactive API like this: 
"{\"foo\": 2}"
See this
f
Will give this a look ... about to head out for the holiday weekend. Have a great weekend!
k
You too! 🙂
f
@Kevin Kho I am not certain how exactly to run the mutation and what I have set incorrectly. I am getting an error when running the below
Copy code
mutation {
  set_flow_group_default_parameters(input: { 
  flow_group_id: "fe8d69d6-f3c1-4d49-bfcd-7faca514a283", 
  parameters: {mapped_run_name: ["DEV_push1"]}
  }) {
    success
    error
  }
}
k
Will try this in a bit
f
Thanks! 🙏
k
Can you try:
Copy code
mutation {
  set_flow_group_default_parameters(input: { 
  flow_group_id: "d412b313-3776-4682-867f-4d89c0679e4f", 
  parameters: "{\"mapped_run_name\": [\"DEV_push1\"]}"
  }) {
    success
    error
  }
}
Replace with your ID
f
Still an exception
Copy code
Exception has occurred: ClientError
400 Client Error: Bad Request for url: <https://api.prefect.io/graphql>

The following error messages were provided by the GraphQL server:

    GRAPHQL_PARSE_FAILED: Syntax Error: Expected :, found String ": ["

The GraphQL query was:

    mutation {
              set_flow_group_default_parameters(input: { 
              flow_group_id: "fe8d69d6-f3c1-4d49-bfcd-7faca514a283", 
              parameters: "{"mapped_run_name": ["DEV_push1"]}"
      }) {
                success
                error
      }
    }

The passed variables were:

    null

The above exception was the direct cause of the following exception:

  File "/workspaces/prism.etl.prefect-flows.db-ingest/db_ingest/flows/ENVERUS_DATA_INTEL_PUSH_FRONTEND/test_graphql.py", line 139, in <module>
    ret_val = client.graphql(update_query)
k
Are you doing this through the UI or Python code?
f
this is directly from python using 
client.graphql(update_query)
k
Oh I see ok, will try from Python. The one above was for the UI
f
I appreciate it
k
Can you try something like this:
Copy code
from prefect.client.client import Client
import json

client = Client()


query = """
mutation($params: JSON!) {
  set_flow_group_default_parameters(input: { 
  flow_group_id: "d412b313-3776-4682-867f-4d89c0679e4f", 
  parameters: $params
  }) {
    success
    error
  }
}
"""

params = {
  "mapped_run_name": ["DEV_push1"],
}

client.graphql(query, variables={"params":params})
f
That got things a little further. Now I get this
Copy code
{'data': {'flow': [{'id': 'fe8d69d6-f3c1-4d49-bfcd-7faca514a283'}]}}
{'data': {'set_flow_group_default_parameters': {'success': False, 'error': None}}}
Ok ... went to the UI and grabbed the "Flow Group ID" from there. Now I see success : True
@Kevin Kho It looks like I am good now. Thanks for showing me how to do this!
k
Nice! No problem!
f
Hey @Kevin Kho I managed to update the flow default parameters but running into a bit of a nuance. For a given flow I ran into the following • Registered a flow with a schedule/clock without parameters • Ran the graphql to update the flow default params • Let the schedule run and it had the params ... Good • Future scheduled times have the params set .... hmm ok • Re-ran the the graphql to update the flow default params with different params .... The future scheduled times have the OLD params set ... not what I wanted to happen Is there a way to refresh any and all existing schedules at once to use the default parameters?
n
Hi @Felipe Saldana - I think the issue here is that those future runs were likely created before you updated the default params. The quickest way to refresh those would be to toggle the schedule off and back on again, which will create the runs with your new defaults. Another option is to add default parameters by schedule, which can be done through the UI, overriding the ones you need but falling back to your set defaults otherwise
f
Thanks @nicholas I will try toggling off/on the Flow schedule tomorrow morning. I am currently testing out just using GraphQL to set the clock + params at once too
Copy code
query = """
    mutation($flow_group_id: UUID!,$cron_clocks: String!, $params: JSON!) {
        set_flow_group_schedule(input :{
            flow_group_id: $flow_group_id,
            cron_clocks: [{cron: $cron_clocks, parameter_defaults: $params}],
            timezone: "UTC"
        }) {
            success
            error
        }
    }
  """
n
That works as well!
Let us know how it goes 🙂
f
will do!
4 Views
Open in Slack
PreviousNext
Powered by