Bennett Rand
04/03/2023, 10:10 PMboto3
in Prefect flows?
I have a task that I'm iterating over an input, and if I map it or .submit() -> .result()
in a for loop for each item, certain boto calls take upwards of 3 minutes to resolve on ECS (not locally).Bennett Rand
04/03/2023, 10:10 PMlogger = project.utils.logging.get_run_logger()
logger.debug("Got Logger")
session = boto3.session.Session()
logger.debug("Got Session")
s3 = session.client("s3")
logger.debug("Got S3 client")
paginator = s3.get_paginator("list_objects_v2")
<http://logger.info|logger.info>("Getting data set=%s", set)
Lines 5 and 7 are taking ~3 minutes eachNate
04/03/2023, 10:42 PM.submit() -> .result()
fits into the picture here? is the above within a task that's being mapped?Bennett Rand
04/03/2023, 10:43 PMstation_futures = get_stations.map(FACTORIES, bucket_name)
for f in station_futures:
f.wait()
and
results = []
for f in FACTORIES:
fu = get_stations.submit(f, bucket_name)
results.append(fu.result())
Both have a delayBennett Rand
04/03/2023, 10:45 PMresults = []
for f in FACTORIES:
results.append(get_stations(f, bucket_name))