Tom Klein

07/06/2023, 9:12 AM
hey there, I have a general question about “best practices” - we are trying to use Prefect to load data into the AWS Sagemaker feature store. Unfortunately the
func there is sync (the batch “ingest” is not working for us right now but that’s not important). We thought of using Prefect to parallelize the
invocations by wrapping each one with a @task , but since we have tens of thousands of records, this would yield tens of thousands of tasks and in practice, it seems that we can’t even get to the point of them running, since over 10+ minutes are just spent submitting these tasks - and they’re all pending…. is this the wrong approach?