I’m wondering if it’s possible for Prefect to run ...
# prefect-server
j
I’m wondering if it’s possible for Prefect to run the same Flow at the same time with different Parameters? Let’s say i have a flow call Balance and i want to run two or more instances at the same time but with different input Parameters, is that possible?
k
Yep see this. It just doesnt scale to more than 10
j
oh awesome! I’ll go read up on it. Thanks for the info
And do you mean it doesn’t scale to more than 10 flows running at once?
k
Yes exactly. At that point, just make a new Flow with new default parameters
j
Okay awesome, is there any plans for the future to remove that limit?
z
Note this limit is for schedules with varying parameters, you can create as many flow runs as you want that have different parameters
k
No plan. It’s the way the scheduler works that it scheduled the next 10 flow runs so if you have 13 varying parameters, 3 of them don’t get scheduled for a given flow
z
For example, without a schedule you can create arbitrary parallel flow runs with different parameters
parallel --jobs 30 prefect run --name "hello-world" --project "example" --execute --param "hello={}" ::: {1..30}
j
Okay, just so i understand correctly. If my flow has 12 tasks and it takes 5 parameters. One of those is host / server. The flow is generic and does stuff based on the host parameter. We can start an unlimited amount of this flow in parallel just by giving different parameters correct?
z
Yes
🙌 1
j
Is it possible to do that by simply calling
Copy code
flow.run()
Or do you ahve to do it via the cli?
k
I think you need a backend to run multiple runs concurrently.
flow.run()
is not intended for production use cases
j
What is intended for production use cases then? How do you send the flow off to get run?
Do you use the apollo api graphql server?
k
You register it against prefect cloud and run it with an agent yep
j
Does flow.run not respect the run_config that’s set on it? Or another way to ask is, does flow.run just run locally always??
I had thought that flow.run would schedule the flow to be run and the agent will pick it up and start running the flow?
k
Yes flow.run does not respect runconfig and storage
Flow.run does not schedule the flow. It’s Prefect Cloud that does when you register it, and then the agent picks it up and starts running it
j
We are using the helm chart currently in a k8s cluster. We aren’t using the Prefect Cloud currently. Is the only way to start or schedule a job through the UI?? There’s no programmatic way to do it?
z
prefect run
will schedule a run if you exclude the
--execute
flag
You must • Register your flow with the server If your flow has a schedule attached, runs will be scheduled automatically. Otherwise • Create runs for your flow via the API, CLI, or UI Then, to submit scheduled runs for execution, you need to • Run an agent
j
Okay sounds good. Is there documentation on how to initiate a flow run via the API?
There’s also the Python client and other methods in that same page
j
Oh awesome! Does
create_flow_run(
return immediately or after the flow has started?
z
Immediately
You can schedule runs for arbitrary datetimes
j
Oh that’s incredible news.
Thanks for all the help!