https://prefect.io
Join Slack
Hello! I have a quite high level design question that I don't know how to handle correctly. I think ...
# ask-community
a
Hello! I have a quite high level design question that I don't know how to handle correctly. I think I need some advice <3 Let's say that I have this 
Source
class (which has a 
refresh()
method) that I want both to be able to call through a CLI and within Prefect. Does it make sense to do something like this?
Copy code
class Source():
  
  def refresh(self):
    self.extract
    self.load()
    self.transform()

  @task()
  def extract(self):
    ...

  @task()
  def load(self):
    ...

  @task()
  def tranform(self):
    ...

@flow
def refresh_source():
  source = Source()
  source.load()
This way, I'm able to instanciate an instance of the 
Source
class elsewhere without relying on Prefect. But in the meantime, I find weird to "infect" everything using 
@task
. What do you think?
n
hi @Adrien Besnard - we generally encourage decorating "pure" functions as opposed to instance methods what do you think about inverting this to make it functional?
Copy code
class Source:...

@task
def extract(): ...

@task
def load(): ...

@task
def transform(): ...

@flow
def refresh_source(source: Source):
    extract(...)
    load(...)
    transform(...)
that way (if you want) you can define modular (prefect-unaware) logic elsewhere and call it from the prefect tasks, where 
Source
is just a serializable / portable dataclass one reason I'd recommend this is because a nice advantage of tasks is cache policies and you'd have to explicitly omit 
self
from consideration (as far as computing cache keys) if your tasks are instance methods, and then there's our somewhat subjective preference for functional style 🙂 but if you are stuck with the 
class Source
paradigm, I'd just define something that looks like
Copy code
@flow
def refresh_source(source: Source):
and call that from within 
Source.refresh
- I just wouldn't recommend decorating instance methods if you can help it
a
Thanks a lot for your answer @Nate! I get the pure function paradigm, but I think even with that my question remains: I'll have to write the logic (i mean the sequential invocation of extract, load and transform) twice: • Once within a Prefect flow (to nicely display the tasks within the UI) • Once within a classic function (let's say a @click.command()) That's the repetition I'm trying to avoid actually.
(I hope I'm clear: I'm quickly writing from my phone, sorry for that)
n
hmm i don't understand this part
Once within a classic function (let's say a @click.command())
why not just call the flow you already wrote here?
a
Because in some circonstances, I need to able to run the source refresh directly from the CLI but without leveraging on Prefect for the execution
Or maybe you mean doing something like refresh_source.fn()?
n
Copy code
» ipython

#[1]
from prefect import flow

#[2]
@flow
def foo(): print('hi no prefect pls')

#[3]
foo.fn()
hi no prefect pls
yeah
a
Okay, fair enough, I can do that
n
or another option (which we do in library land often to keep APIs fungible but non-breaking)
Copy code
@flow
def public_foo():
  _private_foo()
but i think that's a situational preference thing (i personally don't like arbitrary extra wrapping unless we really need it)
a
To give you more context: we have a python project with all our flows defined, and that package depends on other packages that do not depends on Prefect. In the end, a flow is "just" a passthrough to a function defined in one of the other packages
In the end, none of our flows have tasks because the logic is already embedded in the other packages
👍 2
n
that makes sense, my second suggestion wouldnt be as good then bc you'd to dynamically make tasks
a
What do you mean by "dynamically make tasks" ?
n
im assuming that youd have some function 
foo
that has (normal non-prefect) functions inside which you'd want to "be tasks" when you want 
foo
to run as a flow, but since > none of our flows have tasks because the logic is already embedded in the other packages those task definitions don't exist yet
a
Yes exactly
And I wonder if it make sense to actually make the project containg those function dependent from Prefect anyway and adding the @task
It will still be able to run "standalone", but if used within a flow, it will be well integrated with Prefect
(I have the exact same question with the artifacts, actually)
(In the sens that we have to choose if we're okay if the Prefect API "leaks" inside our internal librairies)
n
hmm yeah this has come up before - I will think about this!
9 Views
Open in Slack
PreviousNext
Powered by

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.

Powered by