Kelvin DeCosta
12/21/2022, 11:42 PM.map
for all of them (like the daredevil that I am) with a limit of 20 at a time but it started hanging more often. Right now I’ve chosen to create batches of 20, run one batch of tasks concurrently and then move on to the next batch.
• Deployments are run via ECS Tasks with 4 vCPUs and 8 GB memory
I’m aware of the ongoing concurrency refactor and I’m excited for it, but I want to build something reliable right now.
I’d really appreciate it if I could get some of the following questions answered:
• Would using async
improve the reliability of the tasks?
• Will explicitly creating a new ConcurrentTaskRunner
help?
• What do you think is the overhead that causes the tasks to take much more time than what feels necessary?
I’m open to any suggestions and any help is really really appreciated!
Thanks 😊Zanie
12/22/2022, 12:02 AMKelvin DeCosta
12/22/2022, 12:09 AMZanie
12/22/2022, 12:12 AMKelvin DeCosta
12/22/2022, 9:32 AMpersist_result=False
for the tasks that were running long and it seemed to reduce the run duration by 10-20%.
Edit: I was getting some errors with async
, but I seem to have resolved themYaron Levi
12/22/2022, 11:57 AMpersist_result=False
?Kelvin DeCosta
12/22/2022, 12:01 PMasync
, await
because it seems to be much more smoother.
Some points to consider:
• have tested it only locally so far, but previously my sync mapped tasks couldn't even run on my computer
• persist_result
might not be necessary, since most of the overhead seems to be the context switching and dispatching of threads for sync tasks, which isnt the case for async
• In my case, I needed the State
(to check whether there was an error and not to immediately fail the flow), so I had to use asyncio.gather
to consume the resultsYaron Levi
12/22/2022, 12:04 PMKelvin DeCosta
12/22/2022, 1:22 PMasync
tasks and batching.
A batch of n
async tasks executes via .map(params, return_state=True
.
Later the states are consumed via asyncio.gather()
.