https://prefect.io
Join Slack
Hi all! new member here, so hoping not to be asking an already answered question! Q: How do you set ...
# ask-community
a
Hi all! new member here, so hoping not to be asking an already answered question! Q: How do you set up a project folder structure that allows locally developing and registering multiple flows? If using a single script (not a package) having
Copy code
if __name__ == '__main__':
    # run/register flow
works fine. But if you have modules python does not like it. I was hoping for a structure like the screenshot attached below. For now, I import the flow I am working on the 
main,py
file and that's the one I use to register my flow and test locally (with docker-compose, local agent and the UI)
k
I have a similar setup, and it works fine for running things locally. I use 
prefect run -p path/to/file/with/flow.py
(different flows should be defined in different files, just pass the file with the flow you want to run). Similarly the 
prefect register
command can be used to register flows, it takes a 
-p
flag as well. https://docs.prefect.io/api/latest/cli/run.html https://docs.prefect.io/api/latest/cli/register.html
a
Copy code
# uk_prefect_flows/projects/elexon/flow.py
from prefect import Flow

from .tasks import define_settlement_date


with Flow("local") as flow:
    set_date = define_settlement_date()
used the 
prefect run
as you said but getting the relative import error..
Copy code
% prefect run -p 
...
    from .tasks import (
  ImportError: attempted relative import with no known parent package
k
make a 
setup.py
file at your project root with the following contents
Copy code
from setuptools import find_packages, setup

setup(
    name='uk_prefect_flows',
    packages=[
        'uk_prefect_flows',
        'uk_prefect_flows.projects',
        'uk_prefect_flows.projects.elexon',
    ],
)
Then do 
pip install -e .
Then update your imports to be fully qualified 
from uk_prefect_flows.projects.elexon.tasks import define_settlement_date
a
ah ok! Thanks! I get how locally this can work 🙂 Later, when I want to create an image for each flow and push (to ECR/dockerhub), how would that work though? Apologies for the multiple qs!
k
Its a similar idea, just install directly (instead of as an editable package). We basically have the same setup, Dockerfile would look something like
Copy code
FROM prefecthq/prefect:0.15.2

WORKDIR /app

COPY requirements.txt /app/
RUN pip3 install -r requirements.txt

COPY setup.py /app/
COPY uk_prefect_flows/ /app/uk_prefect_flows/

RUN python3 setup.py install
We use the same image for all flows. I think they key thing to remember is the difference between the “flow” and the python code within. Once the flow is running, import should mostly work as expected (i.e. ensure its pip installed, etc). But getting the flow registered and running is prefects domain, and it doesn’t work exactly the same. For example, we build the above docker file, and then register the flow using S3 storage. So while the actual flow code technically exists on the docker image, the container actually uses (to some degree) the registered flow payload from S3. But accessory code, imports, utils, etc all fall under the normal python stuff. I am probably explaining this wrong, I don’t fully grasp how it works 🙂
a
Thanks! will give this a shot and let you know if anything <<cleaner>> comes up 🙂
🙌 1
d
if you are using anaconda/miniconda, you can conda install the conda-build package. Then from the base directory of your project you can run
Copy code
conda develop .
this will add the base directory sys.path, making every module in the project available everywhere
a
actually 2 things I needed; quite simple after all if using 
poetry
for managing packages: 1. 
poetry install
installs the current project as well to allow local development 2. in the script that the flow is defined use full imports (not relative)
3 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by