https://prefect.io logo
Title
o

Ogaday

04/07/2020, 10:43 AM
I'm having some trouble with prefect Context. According to the docs, I can use context for time sensitive tasks and backfills (https://docs.prefect.io/core/faq.html#does-prefect-support-backfills). However, I've had a number of problems with this approach. I can't supply a datetime to the context of a flow which is run on a schedule, as that datetime is never recalculated after the initialisation of the flow run. And I can't seem to overwrite the default time values in the prefect context as described here: https://docs.prefect.io/core/concepts/execution.html#context Finally, a weird thing is that the date values (such as prefect.context.today) seems to be a strings, while the datetime values (such as prefect.context.date) are datetime objects. Am I doing anything wrong? I'm on Prefect 0.9.8 In the meantime, I'm going to look at using parameters instead of context.
c

Chris White

04/07/2020, 1:43 PM
Hi @Ogaday - any backfill run will need to be run off schedule. If you are using Core alone, you can call
flow.run(run_on_schedule=False)
with whatever context information you want to pass. In general, mixing ad-hoc runs with scheduled runs is a better experience when you use the Prefect API (either the newly open sourced prefect server, which requires 0.10.0+, or Prefect Cloud). And you are correct - some values of context are datetimes and others are strings; this is to support certain values and types that Airflow users would expect. The full list can be found here: https://docs.prefect.io/api/latest/utilities/context.html
o

Ogaday

04/07/2020, 1:51 PM
Thanks @Chris White - the main problem I'm having is that I can't overwrite the default context, so I can't see how to run a backfill. In the example above, I'd expect the pipeline to print
foo
and
hello
but it prints a timestamp and the date.
c

Chris White

04/07/2020, 1:53 PM
Oh I see! Yes in this case you’ll want to pass context as follows:
flow.run(context=dict(date='foo', my_date=datetime.utcnow(), today='hello'))
Let me know if that does the trick
o

Ogaday

04/07/2020, 5:29 PM
Thanks @Chris White that works perfectly. In that case it looks like the right thing to do is to use the default context items and overwrite them when I want to run the flow off schedule, like the docs/you suggest. I'll also look into the open source Prefect Orchestrator - haven't had a chance since it was announced last week, but it's super exciting news and I'm keen to get to grips with it!
c

Chris White

04/07/2020, 5:30 PM
Awesome! Let us know if you have any more questions, always happy to help
@Marvin archive “How to override Prefect context values for a flow run”