Hey everyone.
I am CTO of a small startup and we w...
# ask-communityi
Igor
09/14/2024, 6:23 PMHey everyone.
I am CTO of a small startup and we want to leverage prefect as a workflow engine. Basically we have a main application, that contains dynamic flows, tasks, and some tasks which must be executed only in the container of specific type (because of AI models environment differences). We are going to use k8s to autoscale pods with tasks and incorporated AI models. However I got a couple of related questions
1. Given the architecture that I pinned - should I use specific TaskRunner (like Ray) or prefect worker\work pools to run containarised model's tasks?
2. How to run specific task on a specific container type (task for AI model #1 should not be run on container with AI model #2). I didn't find answer to this question. As I understood workers and work pools have a type to determine which flow to run in container, not the task
3. Can you suggest how should I incorporate prefect into such architecture, maybe give me a set of prefect components I should use to create a such system?
Thank you in advance!🎉
👀 1
t
Taylor Curran
09/14/2024, 6:26 PMHi Igor, to have a part of your flow execute in its own container, you can use subflows or more specifically sub-deployments — the 3rd option listed in this blog post Flow of Deployments
Taylor Curran
09/14/2024, 6:27 PMtasks execute in the same container as their parent flow (unless you use dask to send execution to a remote dask cluster)
Usually flow of deployments are used when you need a sub part of a flow to execute in a different environment
Taylor Curran
09/14/2024, 6:30 PMA helper function that makes is easy is run_deployment
Copy code
from prefect.deployments import run_deployment
run_deployment(
name="my-first-flow/my-first-deployment",
parameters={"my_param": "42"},
job_variables={"env": {"MY_ENV_VAR": "staging"}},
timeout=0, # don't wait for the run to finish
)Taylor Curran
09/14/2024, 6:31 PMYou would have 2 work pool types, one ECS and one k8s
Your parent deployment would point to the ECS work pool
You’re sub-deployments would point to the k8s work pool
Taylor Curran
09/14/2024, 6:37 PMSome users even like to wrap these run_deployment calls inside of tasks to get hybrid behavior
https://github.com/taylor-curran/prefect-patterns/blob/main/flows/composite_flow_patterns/task_wrapped_deployments.py#L18
i
Igor
09/14/2024, 6:41 PMHi @Taylor Curran! Thank you for your response
Is there any way to run only task in a separate container? The reason why I ask this is because container API typically has only one function exposed, don't know is this ok to wrap it with flow (so basically there would be a flow with just one task call inside). Or is there any example of using Dask or Ray to meet such criteria?
t
Taylor Curran
09/14/2024, 6:42 PMI wonder if you might be interested in our new in Prefect 3, background task or task workers?
https://docs-3.prefect.io/3.0/develop/deferred-tasks#run-tasks-in-the-background
i
Igor
09/14/2024, 6:45 PMAh, no, unfortunately. These tasks needs to be executed real time and result of execution must be obtained
Igor
09/14/2024, 7:08 PM@Taylor Curran is this possible also to execute such subflows in parallel? For example if we have 10 containers of specific type and I submit 100 items (like using .map function on task), will these flows be executed in parallel on 10 containers?
t
Taylor Curran
09/16/2024, 2:18 PMTask workers are pretty new — I’m not sure, I’d start a new thread in #CL09KU1K7 with this question 🙂 Thanks 🙏
🙌 1
7 Views