Is is possible to get a subflow to use a specific ...
# ask-communityg
Gregory Hunt
08/31/2023, 5:50 PMIs is possible to get a subflow to use a specific deployment configs?
Gregory Hunt
09/01/2023, 12:15 PM@Nate
n
Nate
09/01/2023, 1:14 PMhmm can you give an example? im not sure what you mean
g
Gregory Hunt
09/01/2023, 1:15 PMI have a deployment with a flow, in that flow is a subflow
Gregory Hunt
09/01/2023, 1:15 PMthe subflow is not associated to any specific deployment
Gregory Hunt
09/01/2023, 1:16 PMbut it does spin up another job, this is in k8s by the way
Gregory Hunt
09/01/2023, 1:17 PMhow do I set parameters for that subflow job/container
n
Nate
09/01/2023, 1:19 PMare you saying that somehow the subflow process is happening somewhere other than the parent flow run? or that the subflow is triggering a k8s job via the API on a cluster somewhere else
Nate
09/01/2023, 1:22 PMbecause in general, deployments (their work pools) are the way for prefect to allow for infra configuration
g
Gregory Hunt
09/01/2023, 1:32 PMYes the subflow creates a secondary container
n
Nate
09/01/2023, 1:35 PMso it sounds like that job creation is not via prefect, so I'd guess you'd have to configure that job through the API thru similar means to how you created it
g
Gregory Hunt
09/01/2023, 1:35 PMno it is via prefect
n
Nate
09/01/2023, 1:35 PMah sorry, can you show code?
g
Gregory Hunt
09/01/2023, 1:36 PM Copy code
@flow(name="Files from Main Query Flow")
def pipeline(msg: dict):
logger = get_run_logger()
for attr, value in msg.items():
<http://logger.info|logger.info>(f"{attr}, {value}")
if msg["status"] == "MALWARE_SCAN_COMPLETE":
parse_provider_data.pipeline(msg["storage_path"])Gregory Hunt
09/01/2023, 1:36 PM Copy code
@flow(name="Parse Provider Data from CCDA")
def pipeline(filename: str):
logger = get_run_logger()
<http://logger.info|logger.info>(f"Parsing: {filename}")
sleep(randint(1, 10))Gregory Hunt
09/01/2023, 1:37 PMthe Parse Provider subflow get spun up in another container
n
Nate
09/01/2023, 1:38 PMhow? by default, subflows are run as a subprocess of the parent flow
g
Gregory Hunt
09/01/2023, 1:39 PM Copy code
@flow(name="Parse Provider Data from CCDA")
def pipeline(filename: str):
logger = get_run_logger()
<http://logger.info|logger.info>(f"Parsing: {filename}")
sleep(randint(1, 10))Gregory Hunt
09/01/2023, 1:39 PMsee how it shows up in the ui
n
Nate
09/01/2023, 1:44 PMi may be misunderstanding something - i see that your
Parse Provider Data from CCDAthe Parse Provider subflow get spun up in another containerparse_provider_data.pipeline(msg["storage_path"])g
Gregory Hunt
09/01/2023, 1:45 PMNope, all I am doing is that call
n
Nate
09/01/2023, 1:46 PMthen i would expect that flow is running as a subprocess of
Files from Main Query Flowthe Parse Provider subflow get spun up in another container?
g
Gregory Hunt
09/01/2023, 2:00 PMOk looks like I was wring about this, I saw similar named containers, but with the adjectives and animal names they looked the same
Gregory Hunt
09/01/2023, 2:00 PMIs there a reason to use a subflow over task?
n
Nate
09/01/2023, 2:28 PMgotcha, I'd say it depends:
• use a subflow if you want to call other subflows or tasks
• use a task if you wanna leverage the task runner or do caching