Jacques
04/17/2020, 2:34 PMJeremiah
04/17/2020, 2:36 PMmap
operator to process them in parallel.Jacques
04/17/2020, 2:36 PMJeremiah
04/17/2020, 2:40 PMwith Flow("test"):
all_urls = retrieve_all_urls()
load_result = load_data.map(all_urls)
transform_data.map(load_result)
Jacques
04/17/2020, 2:42 PMJeremiah
04/17/2020, 2:43 PMprefect.client.Client.create_flow_run()
method, passing your payload to the flow as a parameter. We don’t usually encourage this as a first cut because you lose a little visibility since your work is spread across multiple flows, but it may solve the issue and sometimes thats more important 🙂Jacques
04/17/2020, 2:46 PMJeremiah
04/17/2020, 2:46 PMJacques
04/17/2020, 2:46 PMJeremiah
04/17/2020, 2:47 PMJacques
04/17/2020, 2:48 PMJeremiah
04/17/2020, 2:48 PMJacques
04/17/2020, 2:49 PMJeremiah
04/17/2020, 2:50 PMJacques
04/17/2020, 2:50 PMJeremiah
04/17/2020, 2:51 PMJacques
04/20/2020, 2:11 PMKyle Moon-Wright
04/20/2020, 5:58 PMflow.run
to be non-blocking, you should use concurrent.futures
or a threadpool to run the flow.
https://docs.python.org/3/library/concurrent.futures.htmlJacques
04/20/2020, 6:08 PM