Hi, I have 2k shell commands to run - I divide the...
# ask-community
o
Hi, I have 2k shell commands to run - I divide them to batches of 8 commands and launch a flow(similar to `process_pokemon_batch`in this orchestrator recipe) for each--> 250 flows, each running 8 commands. Now I want each flow to run the 8 commands in parallel, each one using it's own CPU. It works great when I test it locally with
concurrent.futures.ProcessPoolExecutor
or
multiprocessing.Pool
but get stuck on the ECS agent. You could think of it as if the `get_pokemon_info`is a CPU bound command that takes 1 minute to run and therefore it makes sense to run these in parallel (not like the current
get_pokemon_info
, which is i/o bound and is a pretty light API call and therefore makes sense to run concurrently using async). So my question is how can I enable my own parallelism? (Great comment about Concurrency versus parallelism is in the Prefect docs)