Siddharth Singh
10/15/2020, 6:01 AM.map
flows work parallel-ly.
We're using prefect local to test this before we decide on buying cloud. But in local I see my functions are evaluated serially.
# Generate Dates
date_generator = generate_keys_for_nppes_urls()
# Download ZIP Files, Instead of downloading files parallel-ly this function is executing first October, then September and then August...
zip_files = download_nppes_data_for_time.map(date_generator)
# Launch NPPES Spark Job
spark_res = launch_nppes_spark_job.map(zip_files)
# Launch NPPES Configuration Job
config_job = nppes_configuration_job()
config_job.set_upstream(spark_res)
Raphaël Riel
10/15/2020, 12:59 PMZanie
10/15/2020, 1:47 PM