Channels
Just a simple question. Can I override `yesterday`...
# ask-communitya
Andor Tóth
03/08/2021, 5:09 PMJust a simple question. Can I override
yesterday Copy code
with prefect.context(yesterday="2021-01-01"):
flow.run(run_on_schedule=False, executor=LocalExecutor())d
Dylan
03/08/2021, 5:12 PM
Hi @Andor Tóth!
I would recommend using a Parameter with a default that you can overwrite: https://docs.prefect.io/core/concepts/parameters.html
a
Andor Tóth
03/08/2021, 5:13 PMBut Parameters can't have a dynamic value as a default (i.e. yesterday)
Andor Tóth
03/08/2021, 5:14 PMthe default can't be a callable, and it must be JSON serializable
d
Dylan
03/08/2021, 5:14 PM
However, you can overwrite context values in Flow Runs from the UI
Dylan
03/08/2021, 5:14 PM
Sure! But you can have a task respond to a static input
a
Andor Tóth
03/08/2021, 5:15 PMbut I can't override the context from the CLI?
Andor Tóth
03/08/2021, 5:16 PMas you've said Context is better, because it can be specied on the UI
Andor Tóth
03/08/2021, 5:16 PMhow should I do that with context?
d
Dylan
03/08/2021, 5:16 PM
You can also overwrite parameters in the UI
a
Andor Tóth
03/08/2021, 5:17 PMbut, from CLI one can't override certain values?
d
Dylan
03/08/2021, 5:17 PM
For example:
Copy code
import pendulum
date = Parameter(name="date", default="yesterday")
@task
def generate_date(date_parameter):
if date_parameter == 'yesterday'
return pendulum.now().subtract(days=1)
return pendulum.parse(date_parameter)a
Andor Tóth
03/08/2021, 5:18 PMokay, I get that
Andor Tóth
03/08/2021, 5:18 PMbut I don't get the thing with context
Andor Tóth
03/08/2021, 5:19 PMor should I use keys which are not used by default?
d
Dylan
03/08/2021, 5:20 PM
From the Flow Run page -> advanced options you can specify / overwrite configuration options
Dylan
03/08/2021, 5:23 PM
I believe that means the values need to be JSON serializable, however
a
Andor Tóth
03/08/2021, 5:23 PMokay, but I prefer backfilling 365 days from CLI
Andor Tóth
03/08/2021, 5:24 PM Copy code
with prefect.context(...):
flow.run()Andor Tóth
03/08/2021, 5:24 PMshould do the same, no?
d
Dylan
03/08/2021, 5:27 PM
with a parameter it would be:
Copy code
for day in range(365):
flow.run(run_on_schedule=False, parameters={"date": pendulum.now().subtract(days=day).to_date_string()})Dylan
03/08/2021, 5:27 PM
You could also use a date range instead of a specific date, depending on how you’d like to organize your Flow Runs
a
Andor Tóth
03/08/2021, 5:28 PMah, I see
d
Dylan
03/08/2021, 5:28 PM
Copy code
flow.run(parameters={"start_date": "2020-01-01", "end_date": "2021-01-01"})a
Andor Tóth
03/08/2021, 5:28 PMin the meantime, I could make it work with context
Andor Tóth
03/08/2021, 5:28 PMI just had to use a different key
👍 1
d
Dylan
03/08/2021, 5:29 PM
Prefect doesn’t make any specific assumptions about time in your Flows, so you’re free to configure them as you see fit 😄
a
Andor Tóth
03/08/2021, 5:29 PMbut there's a paragraph in the documentation regarding backfills
Andor Tóth
03/08/2021, 5:29 PMwhich suggests to override
todayAndor Tóth
03/08/2021, 5:30 PMthanks for your help
Andor Tóth
03/08/2021, 5:30 PMand patience! 🙂
d
Dylan
03/08/2021, 5:30 PM
That’s because users who come from Airflow generally search for something like “prefect backfill” and are looking for an analogous functionality from Airflow, since Airflow DAGs/Schedules have a hard dependency on time
Dylan
03/08/2021, 5:30 PM
Anytime!
a
Andor Tóth
03/08/2021, 5:31 PMI can't deny, I am stuck with the Airflow mindset
d
Dylan
03/08/2021, 5:33 PM
It’s all good! It takes time to learn new ways of thinking =]
4 Views