https://prefect.io logo
Powered by
Title
f

Felipe Saldana

04/12/2021, 11:36 PM
I see there is a flag to set a parameters file when running a flow. I dont see this flag when registering a flow. Is there a way for me to set a parameters file for a flow so that once registered any run could default to the values from the file? 
Run a flow that is registered to the Prefect API

Options:
    --id, -i                    TEXT        The ID of a flow to run
    --version-group-id          TEXT        The ID of a flow version group to run
    --name, -n                  TEXT        The name of a flow to run
    --project, -p               TEXT        The name of a project that contains the flow
    --version, -v               INTEGER     A flow version to run
    --parameters-file, -pf      FILE PATH   A filepath of a JSON file containing
                                            parameters
    --parameters-string, -ps    TEXT        A string of JSON parameters (note: to ensure these are
                                            parsed correctly, it is best to include the full payload
                                            within single quotes)
I see the Parameters section mentions "...set at flow registration". I would like to provide the values in this section without using the UI though.
k

Kevin Kho

04/13/2021, 2:31 AM
Hi @Felipe Saldana! I don't know if those values can be set out of the UI. I will check and get back to you tomorrow
👍 1
Hi @Felipe Saldana! You can use the GraphQL Api to set these: 
mutation SetDefaultParams($input: set_flow_group_default_parameters_input!){
  set_flow_group_default_parameters(input: $input) {
    success
    error
  }
}
Where 
$input
looks like: 
input: {
  flow_group_id: uuid!,
  parameters: jsonb
}
f

Felipe Saldana

04/13/2021, 2:40 PM
Thanks @Kevin Kho thats great!
Hey @Kevin Kho and others 🙂. Trying to get my code to the finish line. I wanted to see if there were any advancements on the register CLI command with regards to passing params or any other options. As a quick refresher this is ultimately what I would like to do: • define a single flow in a file • register that flow to a dev Prefect project with dev specific parameters • When ready to promote, I want to register that flow to a prod Prefect project with prod specific parameters Issues I am running into • When asking the original question register did not have a param option • Manually setting in the UI is not viable • clocks get me partially there but when I set them in code the dev params will get sent to prod when registering to prod • Could use a convention based StartFlowRun approach: a base flow + [flow_name]_dev with params configured + [flow_name]_prod but this approach just doesnt feel right and will rely on some CI/CD magic to pick flows to register Any other tips, tricks, or lessons learned since I first asked this? Or is GraphQL still the way to solve this? Thanks!
k

Kevin Kho

05/28/2021, 2:37 PM
Hey @Felipe Saldana! there hasn’t been any update to this unfortunately. You can open an issue for it, just so we have a written record for it.
f

Felipe Saldana

05/28/2021, 2:46 PM
Ok, can you show me how to open an issue?
k

Kevin Kho

05/28/2021, 2:52 PM
You have a Github account right? Go to the issues page and then click 
New Issue
on the top right
👍 1
f

Felipe Saldana

05/28/2021, 8:40 PM
@Kevin Kho I am putting together the GraphQL and I am wondering what is meant by jsonb here 
input: {
  flow_group_id: uuid!,
  parameters: jsonb
}
k

Kevin Kho

05/28/2021, 8:56 PM
I think it means it’s stored as a jsonb object, which is json in binary form. You can query the parameters through the interactive API like this: 
"{\"foo\": 2}"
See this
f

Felipe Saldana

05/28/2021, 9:01 PM
Will give this a look ... about to head out for the holiday weekend. Have a great weekend!
k

Kevin Kho

05/28/2021, 9:11 PM
You too! 🙂
f

Felipe Saldana

06/02/2021, 12:01 AM
@Kevin Kho I am not certain how exactly to run the mutation and what I have set incorrectly. I am getting an error when running the below 
mutation {
  set_flow_group_default_parameters(input: { 
  flow_group_id: "fe8d69d6-f3c1-4d49-bfcd-7faca514a283", 
  parameters: {mapped_run_name: ["DEV_push1"]}
  }) {
    success
    error
  }
}
k

Kevin Kho

06/02/2021, 12:03 AM
Will try this in a bit
f

Felipe Saldana

06/02/2021, 12:04 AM
Thanks! 🙏
k

Kevin Kho

06/02/2021, 1:05 AM
Can you try: 
mutation {
  set_flow_group_default_parameters(input: { 
  flow_group_id: "d412b313-3776-4682-867f-4d89c0679e4f", 
  parameters: "{\"mapped_run_name\": [\"DEV_push1\"]}"
  }) {
    success
    error
  }
}
Replace with your ID
f

Felipe Saldana

06/02/2021, 1:12 AM
Still an exception 
Exception has occurred: ClientError
400 Client Error: Bad Request for url: <https://api.prefect.io/graphql>

The following error messages were provided by the GraphQL server:

    GRAPHQL_PARSE_FAILED: Syntax Error: Expected :, found String ": ["

The GraphQL query was:

    mutation {
              set_flow_group_default_parameters(input: { 
              flow_group_id: "fe8d69d6-f3c1-4d49-bfcd-7faca514a283", 
              parameters: "{"mapped_run_name": ["DEV_push1"]}"
      }) {
                success
                error
      }
    }

The passed variables were:

    null

The above exception was the direct cause of the following exception:

  File "/workspaces/prism.etl.prefect-flows.db-ingest/db_ingest/flows/ENVERUS_DATA_INTEL_PUSH_FRONTEND/test_graphql.py", line 139, in <module>
    ret_val = client.graphql(update_query)
k

Kevin Kho

06/02/2021, 1:13 AM
Are you doing this through the UI or Python code?
f

Felipe Saldana

06/02/2021, 1:13 AM
this is directly from python using 
client.graphql(update_query)
k

Kevin Kho

06/02/2021, 1:14 AM
Oh I see ok, will try from Python. The one above was for the UI
f

Felipe Saldana

06/02/2021, 1:14 AM
I appreciate it
k

Kevin Kho

06/02/2021, 1:28 AM
Can you try something like this: 
from prefect.client.client import Client
import json

client = Client()


query = """
mutation($params: JSON!) {
  set_flow_group_default_parameters(input: { 
  flow_group_id: "d412b313-3776-4682-867f-4d89c0679e4f", 
  parameters: $params
  }) {
    success
    error
  }
}
"""

params = {
  "mapped_run_name": ["DEV_push1"],
}

client.graphql(query, variables={"params":params})
f

Felipe Saldana

06/02/2021, 1:36 AM
That got things a little further. Now I get this 
{'data': {'flow': [{'id': 'fe8d69d6-f3c1-4d49-bfcd-7faca514a283'}]}}
{'data': {'set_flow_group_default_parameters': {'success': False, 'error': None}}}
Ok ... went to the UI and grabbed the "Flow Group ID" from there. Now I see success : True
@Kevin Kho It looks like I am good now. Thanks for showing me how to do this!
k

Kevin Kho

06/02/2021, 1:58 AM
Nice! No problem!
f

Felipe Saldana

06/03/2021, 10:50 PM
Hey @Kevin Kho I managed to update the flow default parameters but running into a bit of a nuance. For a given flow I ran into the following • Registered a flow with a schedule/clock without parameters • Ran the graphql to update the flow default params • Let the schedule run and it had the params ... Good • Future scheduled times have the params set .... hmm ok • Re-ran the the graphql to update the flow default params with different params .... The future scheduled times have the OLD params set ... not what I wanted to happen Is there a way to refresh any and all existing schedules at once to use the default parameters?
n

nicholas

06/03/2021, 11:06 PM
Hi @Felipe Saldana - I think the issue here is that those future runs were likely created before you updated the default params. The quickest way to refresh those would be to toggle the schedule off and back on again, which will create the runs with your new defaults. Another option is to add default parameters by schedule, which can be done through the UI, overriding the ones you need but falling back to your set defaults otherwise
f

Felipe Saldana

06/03/2021, 11:11 PM
Thanks @nicholas I will try toggling off/on the Flow schedule tomorrow morning. I am currently testing out just using GraphQL to set the clock + params at once too 
query = """
    mutation($flow_group_id: UUID!,$cron_clocks: String!, $params: JSON!) {
        set_flow_group_schedule(input :{
            flow_group_id: $flow_group_id,
            cron_clocks: [{cron: $cron_clocks, parameter_defaults: $params}],
            timezone: "UTC"
        }) {
            success
            error
        }
    }
  """
n

nicholas

06/03/2021, 11:12 PM
That works as well!
Let us know how it goes 🙂
f

Felipe Saldana

06/03/2021, 11:13 PM
will do!
3 Views
#prefect-serverJoin Slack
Powered by