Hello all, We will have 3 flows for 3 environment...
# prefect-server
i
Hello all, We will have 3 flows for 3 environments (DEV, QA, PROD) ... Those flows are all the same except the AWS resources. Instead of creating 3 different python files, is there a way to pass resource names as a parameter with this command to the python file? prefect register flow --file prefectDemo.py --project \"Data Loading\"
n
Hi @Ismail Cenik - when you say AWS resources, are you meaning the executor and/or run config that you're attaching vary based on environment or are they only parameters in the flow that need to change?
i
Hey, sorry for the confusion. I am using some AWS resources in my flow and I will call their APIs with Boto3. I do not mean the executor. How do I pass their names?
n
Are you running your flows on a schedule? You can pass
default_parameters
to each of your clocks on the schedule
Which would do what you're describing, but I might need to see a small amount of code to recommend better
i
Hello, thank you for the quick response
import time
from prefect import task, Flow from prefect import Client import boto3 XXX_KDA = "...-kda" YYY_KDA = "...-kda" ZZZ_KDA = "...-kda" @task(log_stdout=True) def start_XXX_KDA(): kinesis_client = boto3.client( 'kinesisanalyticsv2) kinesis_client.start_application( ApplicationName = XXX_KDA, RunConfiguration = {})
I want to pass XXX_KDA, YYY_KDA, ZZZ_KDA as a parameter
n
So something like this?
Copy code
from prefect import task, Flow
from prefect import Client
import boto3

@task(log_stdout=True)
def start(application):
    kinesis_client = boto3.client(
        'kinesisanalyticsv2)
    kinesis_client.start_application(
        ApplicationName = application,
        RunConfiguration = {})

with Flow("my flow") as flow:
    application = Parameter("application", default="XXX_KDA")

    start(application=application)
And then when you go to run that, you can set the
application
parameter to whichever application you want?
i
Jenkins job starts it with CLI command
Also I want to pass a parameter like Jira Ticket Id
n
Perfect, can you show which command you're using to run the flow?
i
There are two cases, I do not want to mix. 1) Register flow (with different parameters KDAs). I sent the comment 2) Jenkins job starts CLI Run Flow command, maybe like this
Copy code
prefect run cloud --name "My Flow Name" --project "Hello, World!"
You can guide me for the specific command. The only requirement, I want to pass a parameter which is Jira Ticket Number
n
Copy code
prefect run cloud --name "My Flow Name" --project "Hello, World!" --parameters-string '{"ticket_number": <<ticket_number_from_jenkins>> }'
and in your code:
Copy code
with Flow("My Flow Name") as flow:
    ticket = Parameter("ticket_number", required=True)
    # pass ticket to whatever tasks need it
Does that make sense?
i
That's great for item 2. Thank you
👍 2
For item 1, do I have to create separate Python Files for each environment?
n
You shouldn't need to, you should be able to use a different parameter for that as well; if you want it to be 3 different flows, you can register them under different names and then pass the name in the
run
command differently?
i
In this case, for the scheduled run, I need to pass through those parameters as default parameters. For the Prefect-UI, we need to also provide those parameters where we will not use Prefect-UI for job triggers.
n
Would varying parameter clocks satisfy your first use case?
i
We will try, thank you.