Looking for solutions on how to "call a task from within a task"
• My main task, "Get data from sql" will get the data.
• Within the above function, I run another function to generate the SQL-statement with the specifics sent into the parent function (such as a date from a list of datetimes).
• Since I cannot decorate the sql-generating function with a @Task - how can I run it and still see it in Prefect?
I see from the docs that my options are either 1) use sql.fn() or 2) not decorate it with a @Task - neither of which will not generate a task run.
Here is a mockup:
@task
def sql_builder(date):
statement = statement generator with date and tables
return statement
@task
def get_data(date):
sql = sql_builder(date)
get_data(sql, con)
✅ 1
t
Tony Piazza
09/12/2022, 12:54 PM
@Stefan why not make the get_data function a flow instead? your flow can call it directly as a subflow.
s
Stefan
09/12/2022, 12:57 PM
Thank for your reply.
My first thought was that would seem pretty overkill to generate that amount of Flow-runs in addition to the main one, especially if I'm doing a year's worth of data (one subflow per day) (imagine the UI)
t
Tony Piazza
09/12/2022, 1:19 PM
curious why do you need to "see it in Prefect" ?
s
Stefan
09/12/2022, 1:29 PM
To take advantage of what Prefect has to offer for tasks? See output, use retries, map-function, tags.
t
Tony Piazza
09/12/2022, 1:30 PM
then it has to be either a task or flow. will be interested to see how you decide to implement it.
b
Bianca Hoch
09/12/2022, 10:11 PM
Hi Stefan, we wouldn't recommend using
.fn()
due to the fact that you lose all orchestration functionality that prefect tasks have. You can, however, have a flow call the task, and a flow call a flow.
Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.