Joe Schmid
09/12/2019, 6:32 PMwith Flow("ScaleDownDaskOffHours", environment=env, schedule=schedule) as flow:
config = get_dask_scheduler_config()
new_config = reset_config(config)
save_results = save_new_config(new_config)
save_results.set_upstream(None, flow)
ifelse(new_config != config, save_results, None)
I'm getting the warning PrefectWarning: One of the tasks passed to the switch condition has upstream dependencies: <Task: save_new_config>. Those upstream tasks could run even if the switch condition fails, which might cause unexpected results.
I thought the save_results.set_upstream(None, flow)
line would take care of that but clearly not. What's the right way to address that?Chris White
09/12/2019, 7:13 PMflow.visualize()
I’d bet you’ll see a None
Constant task above save_results
, and that is the task being referenced in the warning — what was your intent with running save_results.set_upstream(None, flow)
? Prefect interprets that first argument as something that should be promoted to a Prefect Task, and in this case that task is simply something which returns the value None
Joe Schmid
09/12/2019, 8:02 PMnew_config != config
is TrueChris White
09/12/2019, 8:08 PMsave_results.set_upstream(None, flow)
line and the Flow should do what you expectJoe Schmid
09/12/2019, 8:22 PMChris White
09/12/2019, 8:22 PMJoe Schmid
09/12/2019, 8:28 PMset_upstream
line after getting the warning.) The warning does make some sense to me -- it's basically saying "hey, the save_new_config task has reset_config as an upstream task -- that might get you weird results since you're using save_new_config/save_results in a switch/ifelse so make up your mind -- do you want it to run when reset_config completes or when your ifelse() condition is True!!!"Jeremiah
09/12/2019, 9:01 PMwarnings
module filters (which might be overkill). I think we originally anticipated ifelse
being used to kick off one of two task branches (which is an Airflow-ish pattern), rather than being used to gate progress on a single branch, but as you point out this is clearly a good use case — and an annoying one to be warned about.Joe Schmid
09/12/2019, 10:50 PM