Hi, I'm getting lost in the examples for concurrency because they all seem to be threaded or use Dask. I'd like to under docker today, and k8s later, deploy flow1 that will read some API data in the form of a list of dictionaries, and then trigger some number of flow2 with each of them receiving 1 entry from the list of dicts. I'd prefer that each run it its own container instance, and as its own process not as a thread in the same process / container. I thought it would be something like calling deployment 2 from deployment 1 but I can't seem to find an example... am I missing something? Thanks!
r
redsquare
07/04/2023, 8:01 PM
You can use the create flow runs from deployment and pass in params for each entry
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.