Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Prefect Community

Hi all,
We are looking into Prefect for a data science project. Couple of things are not immediately clear (we have history with Airflow):
1. Is there a concept of a dags/flows folder where Prefect server looks for flow definitions?
2. Is there a possibility on the core UI to authenticate against LDAP?

Hi <@U0116J9RQN8>, Prefect’s model is different from Airflow’s in how flows are handled. Airflow needs a DAG folder because every single time your interact with a DAG its code gets parsed and executed. Prefect works differently - you `register` your flow with the server, which sends all the flow metadata and makes it appear in the UI and become available for action. You use Prefect Core’s `Storage` classes to let the server know where the flows live - for example, you could store it locally in any folder you want, or in Docker, or cloud storage, etc. When it’s time to run the flow, the server sends the instruction to your agent, which loads the flow from storage and executes it. The advantage of this model is you never risk odd side effects from re-executing the DAG file.

In addition, there are some nice things we can do when you register the flow - for example, you can version your flows to automatically archive an old version and replace it with a new version.

Here is the <https://docs.prefect.io/orchestration/tutorial/configure.html|server deployment tutorial>, which will walk you through getting started with registering flows against the server  / Cloud. Let us know if that’s helpful!

In addition, we view auth as out-of-scope for the server at this time as we work toward workflow feature stability. Auth is understandably something people have many questions about, and I wrote a response to a similar issue last night which may give more detail: <https://github.com/PrefectHQ/prefect/issues/2238>.

Please let us know how we can help you further!

Thanks, <@UKTUC906M> for the explanation, I need to adjust myself a bit :slightly_smiling_face: Auth is not critical to us, Prefect would be internal, just a safety net.

great explanation <@UKTUC906M>! thanks. i too am coming from an airflow world and was wondering how this would work. the prefect UI going open source has gotten me completely on the train

<@U0116J9RQN8> what cloud platform are you on?

<@UKTUC906M> Great explanation!
It would be nice if the explanation put on Prefect's docs. I think many Prefect users come from Airflow world and stuck by this concept.
Users with Airflow experience (including me) tend to lookup docs to let server to parse the definition files of flows, just like Airflow does. But in Prefect the concept is totally different and it's hard for the users to solve this problem from docs without knowing this concept.

It may be very helpful if there is one small chapter in docs to clarify the different concepts when implementing workflows with Airflow and Prefect.