new here and trying this vs airflow for a side project! Can I have classes or methods defined in other files that can be referenced as tasks in a prefect flow? For example, lets say I have this file structure:

scrapers
> scraper1
> scraper2
prefect_flow.py

and scraper1/2 have Classes or methods that scrape specific websites. Am I able to leave those defined in those files, while referencing them in my flow.py file? It would look something like this..

import ...

@task 
def scrape_site_one():
    s = Scraper()
    return s.scrape_site('www...')
....

with Flow("update historical pricing") as flow:
    scrape_site_one()

Of course you can! As long as these files are somehow available in your execution environment… For example: they can be pip installed as a module or available in a

scrapers

directory right besides your flow (for example in a Docker image)

Matthias is pretty right here and it depends how you intend to deploy it (Docker or not), but this can work with a local agent and you just specify the working directory

haha yeah I just did a really dumb flow with a simple class defined in another file

Btw if it’s a side project, maybe you are interested in trying Prefect 2.0, which is in technical preview right now. For production though, 1.0 is recommended

It may go into prod, so I'll keep this for now but open to 2.0 in the future 🙂