Question regarding the level of granularity for Flows/Tasks - we’re moving our Flows to FargateTaskEnvironment, and we’re looking at using the Prefect task mapping to split out partitions of work. I know these can be split out by tricks like using the Docker based task, but those tasks would still be part of the Fargate Task that created them - ideally we want to be able to trigger these as either subflows [don’t think that’s available yet?] or as a Fargate Task in it’s own right. Do you know if anyone has done this? We want to utilise the map/reduce as much as possible, and to get best use out of semi-serverless aspect of Fargate, but right now the only way I can think of is essentially manually overriding a @task definition to be a @MyFargateTask. Is there any other method that users have tried?

Hi @Darragh, I'm not aware of anyone having done this yet but would be really interested to see the results if you do give it a try.

Damn. Was really hoping one of you guys would say “yes this is a painful problem, here’s how we solved it” 🙂

We definitely have a non-makeshift task available, check out Prefect’s nifty FlowRunTask: https://docs.prefect.io/api/latest/tasks/prefect.html#flowruntask

oooooh shiny! I had asked before and was told no but I have a feeling I asked the question wrong 🙂 so then this could actually work well for my first question - I map an input data set that then spawns 10 FlowRunTasks, each of which is it’s own Fargate based Flow, and the input/output results and states can be passed between. Only question then - can the success or failure of the “sub” flow effect the state of the FlowRunTask??

@Darragh no, it'll just schedule the runs for you but won't wait for any results 😕

Ok interesting, thanks @nicholas - my corollary then would be, could I take the returned flow ID, and query it’s state?

Interesting idea, I think that could work with some retries and timeouts in place

Cool! I’m assuming I’d query across GraphQL, something like that? Or would there be another method I could use to query within a flow?

I’d recommend a GQL query. Also, if you want the behavior you described, where you wait for the flow to return, you could modify the flow run task to block and query the API every couple minutes until the flow is in a non-running state, then

raise

an equivalent state so your task takes the same state as the run.

You could use the

prefect.Client

graphql method for that, like this:

import prefect
client = prefect.Client()

client.graphql(
    query="query { flow_run_by_pk(id: <flow_run_id>) { id, state } }", token=<api_token>
)

Brilliant, thanks a lot for that guys! No being able to subFlow was a major concern for us, but this should solve it nicely 👍