Channels
pacc-may-31-2023
prefect-ai
pacc-clearcover-june-12-2023
marvin-in-the-wild
data-ecosystem
geo-israel
pacc-june-14-2023
geo-japan
prefect-cloud
ppcc-may-16-2023
prefect-azure
prefect-docker
prefect-recipes
gratitude
geo-nyc
geo-bay-area
geo-boston
geo-london
geo-dc
geo-chicago
geo-berlin
geo-texas
geo-seattle
geo-colorado
prefect-community
data-tricks-and-tips
prefect-aws
prefect-gcp
introductions
find-a-prefect-job
prefect-dbt
random
events
ask-marvin
show-us-what-you-got
prefect-getting-started
prefect-integrations
prefect-contributors
best-practices-coordination-plane
announcements
prefect-server
prefect-ui
prefect-kubernetes
Title
k
Kyle McEntush
08/22/2020, 12:07 AMIs there a way to run functions that are decorated with @task outside of prefect? I'm trying to rapidly develop individual task stages but I'd prefer to not have two functions defined. Currently, calling a function decorated with @task in a jupyter notebook throws a context error.
c
Chris White
08/22/2020, 12:08 AMHi @Kyle McEntush - absolutely. E.g.,:
@task
def my_function(x, y):
return x + y
my_function.run(1, 2) # runs with no Prefect intervention👍 1
✅ 1
k
Kyle McEntush
08/22/2020, 12:08 AMGot it, thanks!
👍 1
c
Chris White
08/22/2020, 12:09 AM@Marvin archive “How to run a Prefect Task without Prefect”
m
Marvin
08/22/2020, 12:09 AMk
Kyle McEntush
08/22/2020, 1:16 AMInteresting. Calling .run() on a task that lives within a class doesn't automatically pass the "self" argument.
Is that intentional?
c
Chris White
08/22/2020, 1:16 AMCan you share a traceback? Did you initialize the class before calling
.run()k
Kyle McEntush
08/22/2020, 1:18 AMfrom prefect import task
class Test:
def __init__(self):
pass
@task
def test_func(self, val):
return val
test = Test()
test.test_func.run(1)Throws
TypeError: test_func() missing 1 required positional argument: 'val'c
Chris White
08/22/2020, 1:19 AMI see - this is not a supported pattern; all arguments to a task need to be capable of being converted to tasks themselves. To achieve the same thing you should do:
from prefect import Task
class Test(Task):
def run(self, val):
return val
test = Test()
test.run(1)k
Kyle McEntush
08/22/2020, 1:20 AMin that setup, can you have multiple functions that can be run?
c
Chris White
08/22/2020, 1:20 AMWhat do you mean?
k
Kyle McEntush
08/22/2020, 1:20 AMfrom prefect import Task
class Test(Task):
def run(self, val):
return val
def run2(self, val):
return val
test = Test()
test.run(1)
test.run2.run(1)oops, second
I presume that won't work for the same reasons as before
c
Chris White
08/22/2020, 1:22 AMWhat are you trying to achieve with this setup?
k
Kyle McEntush
08/22/2020, 1:22 AMThe end goal is to have multiple tasks defined within a class
but it seems that goes against the design of prefect
c
Chris White
08/22/2020, 1:22 AMbut to what end? Why do the tasks need to all be methods of a single class?
k
Kyle McEntush
08/22/2020, 1:23 AMmany of the tasks share common arguments, so I figured a class would be ideal
But within the current structure I could always pass a dict through to mimic the self arg
c
Chris White
08/22/2020, 1:24 AMgotcha; because Prefect supports distributed environments, the only way to share state between tasks is via arguments <-> return values, so yea a dict of data makes sense
k
Kyle McEntush
08/22/2020, 1:24 AMOkay, thanks for the tip
c
Chris White
08/22/2020, 1:24 AM👍 anytime
k
Kyle McEntush
08/22/2020, 1:24 AMappreciate your responsiveness at 6:30pm on a Friday 🙂
c
Chris White
08/22/2020, 1:25 AMhaha 🍻