https://prefect.io logo
#prefect-community
Title
# prefect-community
d

Daniel Sali

04/01/2020, 12:09 PM
Hi all, We are looking into Prefect for a data science project. Couple of things are not immediately clear (we have history with Airflow): 1. Is there a concept of a dags/flows folder where Prefect server looks for flow definitions? 2. Is there a possibility on the core UI to authenticate against LDAP?
j

Jeremiah

04/01/2020, 12:31 PM
Hi @Daniel Sali, Prefect’s model is different from Airflow’s in how flows are handled. Airflow needs a DAG folder because every single time your interact with a DAG its code gets parsed and executed. Prefect works differently - you
register
your flow with the server, which sends all the flow metadata and makes it appear in the UI and become available for action. You use Prefect Core’s
Storage
classes to let the server know where the flows live - for example, you could store it locally in any folder you want, or in Docker, or cloud storage, etc. When it’s time to run the flow, the server sends the instruction to your agent, which loads the flow from storage and executes it. The advantage of this model is you never risk odd side effects from re-executing the DAG file. In addition, there are some nice things we can do when you register the flow - for example, you can version your flows to automatically archive an old version and replace it with a new version. Here is the server deployment tutorial, which will walk you through getting started with registering flows against the server / Cloud. Let us know if that’s helpful!
In addition, we view auth as out-of-scope for the server at this time as we work toward workflow feature stability. Auth is understandably something people have many questions about, and I wrote a response to a similar issue last night which may give more detail: https://github.com/PrefectHQ/prefect/issues/2238.
👍 3
Please let us know how we can help you further!
d

Daniel Sali

04/01/2020, 12:36 PM
Thanks, @Jeremiah for the explanation, I need to adjust myself a bit 🙂 Auth is not critical to us, Prefect would be internal, just a safety net.
👍 1
t

Thomas La Piana

04/01/2020, 3:40 PM
great explanation @Jeremiah! thanks. i too am coming from an airflow world and was wondering how this would work. the prefect UI going open source has gotten me completely on the train
👍 1
p

Preston Marshall

04/01/2020, 4:38 PM
@Daniel Sali what cloud platform are you on?
g

Gary

07/14/2020, 11:54 AM
@Jeremiah Great explanation! It would be nice if the explanation put on Prefect's docs. I think many Prefect users come from Airflow world and stuck by this concept. Users with Airflow experience (including me) tend to lookup docs to let server to parse the definition files of flows, just like Airflow does. But in Prefect the concept is totally different and it's hard for the users to solve this problem from docs without knowing this concept.
👍 1
It may be very helpful if there is one small chapter in docs to clarify the different concepts when implementing workflows with Airflow and Prefect.
2 Views