https://prefect.io logo
Title
j

Jacob Longhurst

03/15/2022, 7:17 PM
I’m wondering if it’s possible for Prefect to run the same Flow at the same time with different Parameters? Let’s say i have a flow call Balance and i want to run two or more instances at the same time but with different input Parameters, is that possible?
k

Kevin Kho

03/15/2022, 7:18 PM
Yep see this. It just doesnt scale to more than 10
j

Jacob Longhurst

03/15/2022, 7:19 PM
oh awesome! I’ll go read up on it. Thanks for the info
And do you mean it doesn’t scale to more than 10 flows running at once?
k

Kevin Kho

03/15/2022, 7:26 PM
Yes exactly. At that point, just make a new Flow with new default parameters
j

Jacob Longhurst

03/15/2022, 7:33 PM
Okay awesome, is there any plans for the future to remove that limit?
z

Zanie

03/15/2022, 8:24 PM
Note this limit is for schedules with varying parameters, you can create as many flow runs as you want that have different parameters
k

Kevin Kho

03/15/2022, 8:25 PM
No plan. It’s the way the scheduler works that it scheduled the next 10 flow runs so if you have 13 varying parameters, 3 of them don’t get scheduled for a given flow
z

Zanie

03/15/2022, 8:29 PM
For example, without a schedule you can create arbitrary parallel flow runs with different parameters
parallel --jobs 30 prefect run --name "hello-world" --project "example" --execute --param "hello={}" ::: {1..30}
j

Jacob Longhurst

03/16/2022, 8:09 PM
Okay, just so i understand correctly. If my flow has 12 tasks and it takes 5 parameters. One of those is host / server. The flow is generic and does stuff based on the host parameter. We can start an unlimited amount of this flow in parallel just by giving different parameters correct?
z

Zanie

03/16/2022, 8:10 PM
Yes
🙌 1
j

Jacob Longhurst

03/16/2022, 8:15 PM
Is it possible to do that by simply calling
flow.run()
Or do you ahve to do it via the cli?
k

Kevin Kho

03/16/2022, 8:23 PM
I think you need a backend to run multiple runs concurrently.
flow.run()
is not intended for production use cases
j

Jacob Longhurst

03/16/2022, 8:35 PM
What is intended for production use cases then? How do you send the flow off to get run?
Do you use the apollo api graphql server?
k

Kevin Kho

03/16/2022, 8:38 PM
You register it against prefect cloud and run it with an agent yep
j

Jacob Longhurst

03/16/2022, 9:26 PM
Does flow.run not respect the run_config that’s set on it? Or another way to ask is, does flow.run just run locally always??
I had thought that flow.run would schedule the flow to be run and the agent will pick it up and start running the flow?
k

Kevin Kho

03/16/2022, 9:29 PM
Yes flow.run does not respect runconfig and storage
Flow.run does not schedule the flow. It’s Prefect Cloud that does when you register it, and then the agent picks it up and starts running it
j

Jacob Longhurst

03/16/2022, 9:37 PM
We are using the helm chart currently in a k8s cluster. We aren’t using the Prefect Cloud currently. Is the only way to start or schedule a job through the UI?? There’s no programmatic way to do it?
z

Zanie

03/16/2022, 9:40 PM
prefect run
will schedule a run if you exclude the
--execute
flag
You must • Register your flow with the server If your flow has a schedule attached, runs will be scheduled automatically. Otherwise • Create runs for your flow via the API, CLI, or UI Then, to submit scheduled runs for execution, you need to • Run an agent
j

Jacob Longhurst

03/16/2022, 9:41 PM
Okay sounds good. Is there documentation on how to initiate a flow run via the API?
There’s also the Python client and other methods in that same page
j

Jacob Longhurst

03/16/2022, 9:43 PM
Oh awesome! Does
create_flow_run(
return immediately or after the flow has started?
z

Zanie

03/16/2022, 9:44 PM
Immediately
You can schedule runs for arbitrary datetimes
j

Jacob Longhurst

03/16/2022, 9:45 PM
Oh that’s incredible news.
Thanks for all the help!