I have a service that receives HTTP requests (Kafka topics in the future) that triggers some tasks in celery, with all bells and whistles expected from a tasking management system (canceling, monitoring, cache, retry). This above description strikes me as a perfectly acceptable job for Prefect, am I correct? I mean, could I replace all that "engineering" above with a Prefect instance and only be concerned with writing the tasks themselves? Or would I be using Prefect in an "incorrect" way or in an "unforeseen" way? Besides that, does the community have any resources or examples on how to achieve that? I'm mostly looking for examples of how to interact with the flows/deployments with API as a service and on best practices and coordination planes.

Hello @William Jamir I would recommend checking out the REST API docs for more info on interacting with our api, your usecases are doable with Prefect as well as you get some additional observability features! Check out our getting started guide to learn more.

also for some more color @William Jamir, yeah I could see Prefect Cloud (or server) as a stand-in for your service you could point things at the endpoints (like create flow run from deployment) that sahil shared above and trigger work when you receive payloads from the outside world deployments are just the combination of a flow-decorated python function and 2 configurations: • storage (where the code lives) • infra (where and how it runs) and so you could configure things like retries, caching at the flow level and use the prefect UI to monitor everything

Hey William, Will here from the Product team at Prefect. We’re actually looking to make some further investments in this direction in the next few months. Would you be willing to chat about your use case with me for about thirty minutes some time next week?

Thanks @Sahil Rangwala, @Nate I much appreciate the links! Don't hesitate to send more resources or tips regarding this! @Will Raphaelson That is interesting to hear! Just give me some more time to gather some requirements and "corner cases" in a proof of concept and I would be more than happy to share some details with you 😊

Fantastic!

hey, been browsing through community and it seems I've got a very similar use case on my hands too it's a backend with dramatiq tasks (celery alternative) which are triggered by http requests and will soon be kafka topics. are there more details around this? I'd love to replace the current background task workers with Prefect!