Billy McMonagle
12/02/2021, 10:35 PMBilly McMonagle
12/02/2021, 10:36 PMwith Flow("My Flow",) as flow:
parameter = get_parameter()
objects = get_objects()
transformed = transform_objects(objects)
write_data_to_s3(parameter, transformed)
copy_data_to_snowflake(parameter, upstream_tasks=[write_data_to_s3])
What I want to happen is to write_data_to_s3
, and then copy_data_to_snowflake
after write_data_to_s3
is finished. However, this code creates a second copy of write_data_to_s3
in my flow schematic, and does not create the relationship I am looking for. Any advice?Kevin Kho
write_data_to_s3
in a variable and then point to that.
with Flow(...) as flow:
a = first_task()
b = second_task()
c = third_task(c_inputs, upstream_tasks=[a,b])
Kevin Kho
with Flow(...) as flow:
a = first_task()
b = second_task()
c = third_task()
c.set_upstream(b)
c.set_upstream(a)
Billy McMonagle
12/02/2021, 10:38 PMsecond_task
and third_task
both accept the same parameter
value?Kevin Kho
Billy McMonagle
12/02/2021, 10:41 PMa
is an input into third_task
, is it necessary to also set BOTH a
and b
as upstream tasks?Kevin Kho
a
will already be built so I think you should only need b
. This is a hard think thoughBilly McMonagle
12/02/2021, 10:42 PMwrite_data_to_s3
without providing required arguments, which felt completely wrong. Trying again and will report back. Thank you!Kevin Kho
with Flow("ex") as flow:
a = first_task()
b = second_task()
c = third_task(a, upstream_tasks=[b])
I got:Billy McMonagle
12/02/2021, 10:47 PMBilly McMonagle
12/02/2021, 10:48 PMKevin Kho