https://prefect.io logo
Title
k

Kevin Otte

03/10/2022, 5:04 AM
new here and trying this vs airflow for a side project! Can I have classes or methods defined in other files that can be referenced as tasks in a prefect flow? For example, lets say I have this file structure:
scrapers
> scraper1
> scraper2
prefect_flow.py
and scraper1/2 have Classes or methods that scrape specific websites. Am I able to leave those defined in those files, while referencing them in my flow.py file? It would look something like this..
import ...

@task 
def scrape_site_one():
    s = Scraper()
    return s.scrape_site('www...')
....

with Flow("update historical pricing") as flow:
    scrape_site_one()
m

Matthias

03/10/2022, 5:12 AM
Of course you can! As long as these files are somehow available in your execution environment… For example: they can be pip installed as a module or available in a
scrapers
directory right besides your flow (for example in a Docker image)
:upvote: 1
k

Kevin Kho

03/10/2022, 5:14 AM
Matthias is pretty right here and it depends how you intend to deploy it (Docker or not), but this can work with a local agent and you just specify the working directory
k

Kevin Otte

03/10/2022, 5:17 AM
haha yeah I just did a really dumb flow with a simple class defined in another file
k

Kevin Kho

03/10/2022, 5:19 AM
Btw if it’s a side project, maybe you are interested in trying Prefect 2.0, which is in technical preview right now. For production though, 1.0 is recommended
:upvote: 1
k

Kevin Otte

03/10/2022, 5:22 AM
It may go into prod, so I'll keep this for now but open to 2.0 in the future 🙂
👍 1