Marcus Hughes
08/01/2022, 5:05 PMsubflow_count
(the number of subflows a flow generates aka breadth of the recursion tree) and levels
(how many chained subflows there are aka the depth of the recursion tree) are large, 25 and 10 here respectively. That's 10^25 - 1 flows, truly absurd with long chains of subflows. When it fails, I get OSError: [Errno 24] Too many open files
. I noticed that the crashed flows don't get marked as crashed but stay with the running
status. I'm not sure there's an easy way for you to resolve that though. On the positive side, my computer didn't crash! At smaller values, e.g. subflow_count=8
and levels=3
, it runs just fine.
My team and I are excited to use Prefect in the upcoming NASA PUNCH mission to prepare scientific data of the Sun. It's really helped make our pipeline more elegant.John Kang
08/01/2022, 7:45 PMStรฉphan Taljaard
08/02/2022, 7:48 PMChris L.
08/03/2022, 10:28 AMNico Neumann
08/03/2022, 2:57 PMKhuyen Tran
08/04/2022, 9:47 PMCole Murray
08/05/2022, 3:13 AMViet Nguyen
08/05/2022, 8:21 AMAlix Cook
08/05/2022, 7:36 PMKhuyen Tran
08/08/2022, 2:15 PMFalk
08/16/2022, 1:29 PMSaad Zaheer
08/18/2022, 4:01 PMale
08/19/2022, 7:17 AMprefect_transfrom==0.1.6
.
Docs che be found here.
This release introduces a new Transform Block, which is now required to handle sensitive information to authenticate on Transform.
Give it a try!
Thanks @Andrew Huang for the guidance on implementing Prefect Block! ๐ale
08/19/2022, 2:00 PMprefect_stitch==0.1.3
has a brand new block too! StitchCredentials
, which can be used to handle Stitch access token by leveraging the Prefect Block concept ๐ฆ ๐ฆ ๐ฆ
Docs can be found here.Jeremiah
Chris L.
08/21/2022, 9:32 AMimport asyncio
from prefect import task, flow
@task
async def print_x(x):
print(x)
await asyncio.sleep(1)
def build_subflow(name):
@flow(name=f"subflow:{name}")
async def subflow(x):
await print_x(x)
return subflow
@flow
async def parent_flow():
futures = [build_subflow(name=x)(x) for x in ["a", "b", "c"]]
await asyncio.gather(*futures)
Anna Geller
prefect-airbyte
collection in Prefect 2.0Ryan Peden
08/24/2022, 3:43 AMJeff Hale
08/25/2022, 4:39 PMPranit
08/26/2022, 6:03 AMModuleNotFoundError: No module named 'prefect.tasks.airbyte'; 'prefect.tasks' is not a package
Have I missed to install any dependency ? I didnt see anything on airbyte/prefect pages.Jeremiah
Ryan Peden
08/30/2022, 4:19 PMJeff Hale
09/01/2022, 1:21 PMJeff Hale
09/01/2022, 7:10 PMAlberto de Santos
09/05/2022, 3:56 PMAnna Geller
Henning Holgersen
09/07/2022, 9:57 AM