Hello team, I am planning to start using prefect f...
# prefect-community
Hello team, I am planning to start using prefect for the workflow management, But I would like to understand if there is an REST api available for prefect core module.?
Hi Guarav. Welcome to Prefect! Prefect Core is for building, testing and executing locally. Our (graphql) API is part of Prefect Cloud - if you're interested in that you could join our wait list for a Prefect Scheduler account.
If you can give a little more detail about what you would want an API for, we can maybe help you think through your use case too.
Hey Jenny, We are planning to create a workflow to manage the kerenel manager for entire infrastructure. So for that we have the individual task which we planned to run using prefect. But along with that we want to graphically view of the flow, manage the states of the flow, re-run the flow from a certain point incase there is failure and other such details
I am not too familiar with prefect cloud but .. can we setup our own prefect cloud infra .
Prefect can definitely help with visualizing flows, re-running flows and managing state etc!
In terms of infrastructure (I think that's what you mean by 'infra', please correct me if I'm wrong!):
We have developed an innovative "hybrid" model for Prefect Cloud - it is probably more accurate to think of Prefect Cloud as "on-prem lite". Prefect Cloud hosts a database and associated services which are responsible for tracking metadata about your Flows and Tasks. The actual execution of your workflows occurs in your infrastructure, and is orchestrated through your Prefect Agent.
upvote 2
Hi @Gaurav Goel — Prefect Cloud’s services receive only metadata for orchestration and visualization purposes; your code and data remain private and execute on your infrastructure. If you’d like to see how our solution works, we can add you to the queue for our Scheduler product program, which is a free version of the full platform.
You can of course build APIs on top /around Prefect’s open source package as well
newbie here. Does this mean that scheduler is not an option with prefect-core? I'm mostly interested in standing up my own prefect server
Hey Dave! Our scheduler service is free but not open source (it’s a hosted orchestration layer). However all execution, code, and data stay on your servers.
Just to add to what Jeremiah said, Prefect Scheduler is the name of our free Prefect Cloud product. If you're interested in running things on schedule, Prefect Core does also have the ability to do schedules.
then i'm not sure what the difference is. they will both schedule my tasks
Yes they will! Prefect Cloud is like a command center for your flows. You'd still use Prefect Core to write your flows but with Prefect Scheduler (as a part of Prefect Cloud) you can deploy to Prefect Cloud, which gives you access to other features such as a UI and graphql API
This probably explains it better than I did!
ok, what you said makes sense. we'd want the UI for sure but hesitant about relying on 3rd-party (even if you all are the awesomest)
first glance, prefect is a great product. we'll continue to evaluate
👍 2
ok.I was going through the above discussions and still it looks like if we want to have the complete setup in our infrastructure ( nothing external ) it is not possible right ?
And regarding the scheduler I was reading there was Dask scheduler which prefect core uses to give us the scheduling facility ? I was hoping to use that
Will it not be possible
And on the other hand with my original query you mentioned that there are some API's for prefect core as well right ? Could you point me to any of those data
Hi Gaurav, good questions! I'm going to see if one of the guys on the west coast can answer, if not I'll get you an answer in the morning
Hi @Gaurav Goel, based on discussions with our customers to date, most users don’t find managing the orchestration layer itself desirable. So, we came up with (and patented) a hybrid execution setup where all flows will run on our customers infrastructure, but the UI, GraphQL API, and scheduling of flow runs is managed on our infra. In this setup, the Dask scheduler (which farms out tasks to workers during a flow run) lives on the customer infrastructure either as part of a long-running Dask cluster or spun up as-needed. If you’re interested in sitting down with us, we’d be more than happy to walk you through it. Here’s a doc you may find helpful: https://docs.prefect.io/cloud/dataflow.html
upvote 1
👍 1
Prefect core also contains a scheduler for flow runs. Attaching a schedule to a flow will cause it to run on schedule when calling ‘flow.run’
Our GraphQL API is included only as a part of Cloud or Scheduler, as it requires a persistence layer
(We use this in lieu of a REST API)
To set up a time to talk with us, please fee free to reach out at hello@prefect.io
Let me know if I can be of more help