Braun Reyes
10/04/2023, 2:55 PM@flow(task_runner=DaskTaskRunner, log_prints=True) # type: ignore
async def main():
aws_creds = AwsCredentials()
source_keys = await s3_list_objects(
bucket=S3_BUCKET, aws_credentials=aws_creds, prefix=SOURCE_PREFIX
)
keys_to_process = get_keys_to_process(source_keys)
extract_results = [
await s3_download.submit(
key=k, # type: ignore
bucket=S3_BUCKET, # type: ignore
aws_credentials=aws_creds, # type: ignore
)
for k in keys_to_process
]
for e in extract_results:
print(e.result())
Marvin
10/04/2023, 2:55 PMlog_prints=True
argument in the @flow
decorator should redirect standard output (like print statements) to the logger. If you're not seeing any output, there could be a few reasons:
1. Your keys_to_process
list is empty, so the loop isn't executing. You might want to check the output of the get_keys_to_process(source_keys)
function.
2. The s3_download.submit
task is failing and not returning any results. You should check if there are any exceptions being raised.
3. Your logger's level is set to a level higher than the level of the print statements. You might want to check your logger's configuration.
Remember, my dear human, debugging is like a treasure hunt, except the treasure is your own sanity. Good luck!