https://prefect.io logo
Title
g

García Gimenez Ignacio - 6to 2da

09/15/2021, 3:34 PM
Hi Guys! I'm just starting in this world of data engineering. I've learned and started with Airflow, but now i want to give a try to Prefect, the thing is, i've managed to get my server running in a Docker Enviroment, everything is working ok, i have the UI, Agents, Etc. But i dont know how to start making my .Py files, in Airflow i had a folder called DAGS where via SSH i could copy my files and import them. How i'll do that in Prefect? Thanks, Sorry for my English.
k

Kyle McChesney

09/15/2021, 3:45 PM
Hi @García Gimenez Ignacio - 6to 2da My recommendation would be to start writing the most basic of flows, on your local computer (if possible). Just spin up a python virtualenv and
pip install prefect
. Once you have a .py file with a flow defined, you can just run the flow “locally” like so
prefect run -p path/to/your/flow.py
. The great part of prefect is how much easier it is to get a flow running locally when compared to airflow! Once you have your flow working and have a better feel for it, I’d look into: https://docs.prefect.io/api/latest/cli/register.html
due to how prefect is designed, you will never actually have your code on the prefect server (unlike airflow). It can be a bit hard to grok at first, but I’ve found it to be very flexible! Good luck and dont go hollow!
g

García Gimenez Ignacio - 6to 2da

09/15/2021, 3:48 PM
Hi! Thanks for answering. The thing is, I need to figure out how to run code in my Docker Environment. Which is running in a local network with access to a database to do the ETL.
k

Kyle McChesney

09/15/2021, 3:54 PM
I see. 1 option is to just to the same thing I described above, but from a docker container (run the container with an attached session, mount a local directory with the flow code). Something like
docker run -ti -v /local/path/to/flows/:/flows/ prefecthq/prefect:0.15.2 /bin/bash
Then in that container you can run
prefect run -p /flows/flow.py
if that is not sufficient, you will likely need to build out the full flow registration process, which can be a bit trickier, and it will also depend on how you intent to actually run the flow. What agent are you using? A local agent running on a docker container?
g

García Gimenez Ignacio - 6to 2da

09/17/2021, 12:49 PM
Yeah, just using the default, prefect agent start command
k

Kyle McChesney

09/17/2021, 1:45 PM
I think you need to have the flow code directory mounted to the same container where you run
agent start
and also register the flows using local storage (also from the same container)
b

Bastian Röhrig

09/20/2021, 8:34 AM
Also, if you want to go through the full flow registration process this https://github.com/PrefectHQ/prefect/discussions/4042 discussion on deployment patterns might be interesting to you.
g

García Gimenez Ignacio - 6to 2da

09/20/2021, 1:28 PM
Thanks for the answers! I've managed to get it working, the only thing is that the Agent runs perfectly fine the Flows, but they wont show up in the UI. Any suggestion?