Hey guys, i have a new use case and i wanted to know if prefect was well suited for this. i have a web app that will request some data processing to the backend via an internal api. To process this data, i want my internal api to trigger a flow run from the graphql api. This flow run would be dynamically build depending on the arguments it receives and it's task is to process the data given in input. I'm sure that prefect can do that. The thing i'm less sure is the way of retrieving those results. If i understood correctly, we can't get results of a flow from the graphql API. I have an idea for now but i'm completely open for other ideas: • make the flow write the results in a temp db. Those results would be associated to an id provided as a flow parameter. Questions: How to notify that the flow succeded ? I know there is a flow_run_view mutation in the graphql api, is looping on this call until there is a "failed" or "success" status ok ?

You can definitely trigger a flow run via an API call. As for the dynamically generated, you may prefer Prefect 2.0 for that

the way of retrieving those results

Could you explain a bit more what you mean by results here? you could add a task at the end of a flow run e.g. sending someone a notification about the final state and info that the flow run finished. If this doesn't help, could you explain your use case purely from a business perspective? this would be helpful to give a more concrete answer

Yes of course. The result here is usually a dataset. The goal is to allow the user to ask for some data available in an external api that we would call from a prefect flow and make the response available to the user. We are doing this with a simple internal api call for now but we want to add the possibility to "chain" API calls by taking the output of one API and use it as the input of an other API (think about data enrichment for example: Call an API to get name of cities that have more than X inhabitant and then call an other API that will add informations about those cities). Using prefect, we could do that on dynamically by chaining multiple "exctract" tasks.

the cities' example helps, thanks for that! How should this data be used in the end? what would you want to do with the end result? send it to someone via email, use it in some ML model or display this data in some report?

For now, this data should be displayed on the web app in a tabular format to the client that requested it when it is available. In the future, we are planning to add some "Transform" options to the end user. It also has to be downloadable in the desire format. That's why i thought about making it available in a db.

Yup, your intuition is 100% correct. Regardless of how you want to make this data available to the end-user (via a BI tool or some custom front-end application) you need to persist it into some backend DB. I have no vested interest in recommending Snowflake, but storing this sort of data there makes a lot of sense for the analytical use case you described - may be actually interesting to you as it discusses exactly the topic you described and Prefect 2.0 has/will have integrations with both Hightouch and Snowflake

Ok thanks ! Do you think Prefect 2.0 is mature enough to handle this in a production env? When i started prefect a month ago you said it wasn't

I'd say if you use the version that we plan to release this Thursday and you don't do anything crazy with it like interacting with the backend API routes directly but rather using the basic building blocks, then yes. And for full transparency, the target date for Prefect 2.0 😛refect2: to officially come out of Beta status is July this year - we can't promise any fixed date or deadline but this is the target

Ok thanks for the infos. I'll see what i can do with that !

I'm almost sure it wasn't me haha 😄 anyways keep us posted and you can follow #announcements or Discourse about the new releases

Ok no problem 🙂 Yes i will, thanks again for your help ! Always appreciate ^^

That may have been me who suggested that. I think that Prefect 2.0 has gotten quality of life improvements that have made it easier to use. Also, there will be a release sometime soon to add more features.