Hi everyone :wave:, Can someone assist with this q...
# prefect-community
n
Hi everyone 👋, Can someone assist with this question please? I simply try to backfill data by passing a datetime parameter https://stackoverflow.com/questions/64029629/is-there-a-way-to-backfill-historical-data-once-for-a-new-flow-in-prefect @Chris White gave an answer, but I can’t get it to work at all.
j
@Newskooler in the example with Context are you passing in
intended_backfill_start_time
and accessing it as
backfill_time
in the task? That looks to be a typo in the answer
n
Hi @josh you are right, which is the correct way to write it in this case?
I wrote
current_time = prefect.context.get("intended_backfill_start_time") or prefect.context.get("scheduled_start_time")
and then have this unchanged like so
flow.run(context={"intended_backfill_start_time": "2020-09-01"}) # will use old timestamp
(it works).
Now what @Chris White is suggesting is to loop over the
current_time
until today, right?
j
Doesn’t matter which you write. Context here is just accessing an item from a dictionary and therefore the names need to match. Yeah next step would be to loop over a period of time and create the runs
n
Okay, so why do i not see all past such flow runs in the UI - if I run it and then register it (with a schedule) 🤔
(My thinking is too much Airflow like I guess, since I see past backfilled runs there, hence my confusion)
j
You should be seeing runs in the UI for each one you create when looping through time. The runs won’t be technically scheduled for a time in the past though because when you create the run you’re saying “Run this right now but perform some computation using data from the intended_backfill_start_time”
Oh also to note: if you’re calling
flow.run
you won’t see them in the UI, that’s for local in process runs. You’re going to want to use the client
create_flow_run
in order to create the flow run in the backend
n
How should I run in order to see them in the UI and to run them 🤔 : )
j
I recommend running through this deployment tutorial which will explain some of these concepts better: https://docs.prefect.io/orchestration/tutorial/first.html#write-flow
👍 1
n
Thanks. I will read now and may ask again later, if that’s okay @josh: )
👍 1