Hi, what is the best way to run "subtasks"? I know...
# best-practices
m
Hi, what is the best way to run "subtasks"? I know i can just use in taskA
Copy code
taskB.run()
but then it's treated as a normal python function (so no logging, and no other prefect's magic is available for this taskB) I managed to workaround this creating a Flow in a Task, something like this:
Copy code
@task
def taskA(config: dict):
#...
    with Flow() as flowB:
        taskB()
#...
    a = flow.runB()
it seems to work - i get logs for flowB and taskB, etc I can visualize both flows using flow.visualize() - on two separate diagrams - but it's all right for me. But is this a correct approach, or should I do this some other way?
a
There is no concept of subtasks in Prefect and in majority of use cases calling tasks from other tasks in considered an antipattern. I would be curious to hear more about your use case and the problem you are trying to solve this way
m
ok, I will try to describe it here, probably today I'm aware that I may be trying to do something stupid 🙂 thank you
a
Not a stupid idea at all, we just don't have a concept of a subtask. The question is definitely valid since a subtask is a reasonable concept you can find e.g. in products such as a to-do list. But in Prefect 1.0, the main concepts we operate with are tasks and flows. In Prefect 2.0 we make this more flexible by introducing a concept of subflows. This way you can view: • tasks as the smallest atomic units, • subflows as a logical grouping of tasks that belong together (e.g. all tasks loading data to your staging area could be put into a subflow and this subflow could be called from a task) • and flows as the main entry point that coordinates all subflows and tasks and can be scheduled and deployed from the
DeploymentSpec
abstraction
if I understood correctly based on your introduction, you are just getting started with Prefect - if that's the case I would recommend starting with Prefect 2.0, you can learn more about the distinction here
m
I will chceck this, but is Prefect 2.0 production ready already?
a
the answer to this question is in the Discourse topic
m
I'm sorry I didn't understand. What does it mean ?
a
I linked the relevant section that answers your question
I can copy paste here if that's your preference 🙂
m
link is enough, thank you
👍 1
a
Can I use Cloud 2.0 in production already? Cloud 2.0 is a beta product, with beta product guarantees. When deciding whether to start using Cloud 2.0 today, you should consider: • your use case i.e. scope, scale, and complexity of your workflows, • your requirements regarding stability and resilience, • your preference regarding whether you want to optimize for addressing current needs or whether you focus on using Prefect long-term. Given the above criteria, there is no one-size-fits-all answer. Here are some points that may help you choose the right Cloud product as a new user. 1. Start with Cloud 1.0 when: ◦ you are already a Prefect user and/or you already have a Cloud 1.0 account, ◦ you are new to Prefect and your team requires user management, RBAC, and multi-tenant orchestration API before July 2022, ◦ your requirements for stability and resilience outweigh the drawbacks of having to migrate to a new platform later, ◦ you prefer a mature, legacy platform over a beta product. 2. Start with Cloud 2.0 when: ◦ you are optimizing for working with Prefect long-term, ◦ you are willing to work with a product that is currently still in a beta status with beta product guarantees (including potential breaking changes), ◦ you are doing a PoC and your team still needs some time to get familiar with Prefect or workflow orchestration in general.
If your reliability, security, or scaling needs are the same as or less than hosting Prefect 2.0 yourself, start with Cloud 2.0 in production.
m
ok, thx