Riley Hun
08/07/2020, 4:01 PMset_upstream
and set_downstream
work?
Here is an ETL flow I created:
with Flow("Thinknum ETL") as flow:
token = token_task(
version=version,
client_id=client_id,
client_secret=client_secret
)
history = history_task(
dataset_id=dataset_id,
version=version,
token=token
)
loaded_dates = loaded_dates_task(
bucket_name=bucket_name,
dataset_id=dataset_id
)
dates_to_load = get_dates_to_load(
already_loaded_dates=loaded_dates,
history=history
)
datasets_to_load = load_to_gcs_task.map(
dataset_id=unmapped(dataset_id),
date=dates_to_load,
version=unmapped(version),
token=unmapped(token),
bucket_name=unmapped(bucket_name)
)
It DOES seem to work fine, but I don't know when or if I should be applying `set_upstream`/`set_downstream`itay livni
08/07/2020, 4:26 PMRiley Hun
08/07/2020, 4:30 PMitay livni
08/07/2020, 4:30 PM