Tom Klein
07/06/2023, 9:12 AMput_record func there is sync (the batch “ingest” is not working for us right now but that’s not important).
We thought of using Prefect to parallelize the put_record invocations by wrapping each one with a @task , but since we have tens of thousands of records, this would yield tens of thousands of tasks and in practice, it seems that we can’t even get to the point of them running, since over 10+ minutes are just spent submitting these tasks - and they’re all pending….
is this the wrong approach?