https://prefect.io logo
Title
f

Florian Guily

05/11/2022, 9:29 AM
Hey guys, i have a new use case and i wanted to know if prefect was well suited for this. i have a web app that will request some data processing to the backend via an internal api. To process this data, i want my internal api to trigger a flow run from the graphql api. This flow run would be dynamically build depending on the arguments it receives and it's task is to process the data given in input. I'm sure that prefect can do that. The thing i'm less sure is the way of retrieving those results. If i understood correctly, we can't get results of a flow from the graphql API. I have an idea for now but i'm completely open for other ideas: β€’ make the flow write the results in a temp db. Those results would be associated to an id provided as a flow parameter. Questions: How to notify that the flow succeded ? I know there is a flow_run_view mutation in the graphql api, is looping on this call until there is a "failed" or "success" status ok ?
a

Anna Geller

05/11/2022, 11:26 AM
You can definitely trigger a flow run via an API call. As for the dynamically generated, you may prefer Prefect 2.0 for that
the way of retrieving those results
Could you explain a bit more what you mean by results here? you could add a task at the end of a flow run e.g. sending someone a notification about the final state and info that the flow run finished. If this doesn't help, could you explain your use case purely from a business perspective? this would be helpful to give a more concrete answer
f

Florian Guily

05/11/2022, 11:33 AM
Yes of course. The result here is usually a dataset. The goal is to allow the user to ask for some data available in an external api that we would call from a prefect flow and make the response available to the user. We are doing this with a simple internal api call for now but we want to add the possibility to "chain" API calls by taking the output of one API and use it as the input of an other API (think about data enrichment for example: Call an API to get name of cities that have more than X inhabitant and then call an other API that will add informations about those cities). Using prefect, we could do that on dynamically by chaining multiple "exctract" tasks.
a

Anna Geller

05/11/2022, 11:49 AM
the cities' example helps, thanks for that! How should this data be used in the end? what would you want to do with the end result? send it to someone via email, use it in some ML model or display this data in some report?
f

Florian Guily

05/11/2022, 11:52 AM
For now, this data should be displayed on the web app in a tabular format to the client that requested it when it is available. In the future, we are planning to add some "Transform" options to the end user. It also has to be downloadable in the desire format. That's why i thought about making it available in a db.
a

Anna Geller

05/11/2022, 12:03 PM
Yup, your intuition is 100% correct. Regardless of how you want to make this data available to the end-user (via a BI tool or some custom front-end application) you need to persist it into some backend DB. I have no vested interest in recommending Snowflake, but storing this sort of data there makes a lot of sense for the analytical use case you described -

this talkβ–Ύ

may be actually interesting to you as it discusses exactly the topic you described and Prefect 2.0 has/will have integrations with both Hightouch and Snowflake
and Prefect 2.0 would be an ideal orchestrator to make such a process operational and callable via an API (you can trigger such data flow via API, schedule, from the UI etc)
f

Florian Guily

05/11/2022, 12:07 PM
Ok thanks ! Do you think Prefect 2.0 is mature enough to handle this in a production env? When i started prefect a month ago you said it wasn't
a

Anna Geller

05/11/2022, 12:19 PM
I'd say if you use the version that we plan to release this Thursday and you don't do anything crazy with it like interacting with the backend API routes directly but rather using the basic building blocks, then yes. And for full transparency, the target date for Prefect 2.0 πŸ˜›refect2: to officially come out of Beta status is July this year - we can't promise any fixed date or deadline but this is the target
When i started prefect a month ago you said it wasn't
Did I really say that? πŸ˜„ do you have a link, I'm curious. I have a feeling I encourage everyone to start with 2.0 directly because things always take longer and people need time to get familiar with a new tool anyway, and big bang migrations also don't work
f

Florian Guily

05/11/2022, 12:22 PM
Ok thanks for the infos. I'll see what i can do with that !
πŸ‘ 1
Well maybe it wasn't you but someone said i should stick to Prefect 1 for now ^^ I'll try to get the message !
πŸ‘ 1
Oh apparently it was before the 24 of march and i can't access to those msg
yeah it was not long after Prefect 2 entered beta stage
πŸ‘ 1
a

Anna Geller

05/11/2022, 12:42 PM
I'm almost sure it wasn't me haha πŸ˜„ anyways keep us posted and you can follow #announcements or Discourse about the new releases
f

Florian Guily

05/11/2022, 12:44 PM
Ok no problem πŸ™‚ Yes i will, thanks again for your help ! Always appreciate ^^
πŸ‘ 1
k

Kevin Kho

05/11/2022, 1:28 PM
That may have been me who suggested that. I think that Prefect 2.0 has gotten quality of life improvements that have made it easier to use. Also, there will be a release sometime soon to add more features.
πŸ™ 1