Are there any known caveats to using standard `boto3` in Prefect flows? I have a task that I'm itera...

Bennett Rand

04/03/2023, 10:10 PM

Are there any known caveats to using standard

boto3

in Prefect flows? I have a task that I'm iterating over an input, and if I map it or

.submit() -> .result()

in a for loop for each item, certain boto calls take upwards of 3 minutes to resolve on ECS (not locally).

Bennett Rand

04/03/2023, 10:10 PM

Copy code

logger = project.utils.logging.get_run_logger()
logger.debug("Got Logger")
session = boto3.session.Session()
logger.debug("Got Session")
s3 = session.client("s3")
logger.debug("Got S3 client")
paginator = s3.get_paginator("list_objects_v2")
<http://logger.info|logger.info>("Getting data set=%s", set)

Lines 5 and 7 are taking ~3 minutes each

Nate

04/03/2023, 10:42 PM

Hi @Bennett Rand - hmm I don't believe there any known issues related to this just thinking out loud: I suppose if your ECS cluster / S3 bucket are in different regions, or if your api calls are getting throttled somehow, that this could cause some latency.. but ~3 minutes seems like quite a lot can you explain how

.submit() -> .result()

fits into the picture here? is the above within a task that's being mapped?

Bennett Rand

04/03/2023, 10:43 PM

Copy code

station_futures = get_stations.map(FACTORIES, bucket_name)
for f in station_futures:
    f.wait()

and

Copy code

results = []

    for f in FACTORIES:
        fu = get_stations.submit(f, bucket_name)
        results.append(fu.result())

Both have a delay

Bennett Rand

04/03/2023, 10:45 PM

But not

Copy code

results = []

    for f in FACTORIES:
        results.append(get_stations(f, bucket_name))

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.