Ryan
01/10/2022, 4:08 AM@task(result=PrefectResult())
makes the output of the task run appear in the UI, except I'm looking to do this with the inputs as well.Kevin Kho
Ryan
01/10/2022, 4:14 AMPrefectResult
gives us the output we'd like, but I'm struggling to show the associated input in the UI as well.Kevin Kho
Ryan
01/10/2022, 4:21 AMKevin Kho
Anna Geller
Ryan
01/10/2022, 1:27 PM@task
def my_adder_step(a, b):
Parameter(a)
Parameter(b)
result = a + b
return result
?Anna Geller
from prefect import task, Flow, Parameter
@task(log_stdout=True)
def plus_one(x):
print(x + 1)
with Flow("parametrized_flow") as flow:
param = Parameter("x", default=1)
plus_one(x=param)
Ryan
01/10/2022, 1:40 PM@task
def get_urls():
response = requests.get("<http://homepage.example.com>")
urls = response.json()["urls"]
return urls
@task
def classify_webpage(url): # <-- `url` is the thing we want in the UI
response = requests.get(url)
return _is_spam(response.text)
with Flow("spam_detector") as flow:
returned_urls = get_urls()
classify_webpages.map(returned_urls)
The idea is we don't know what URLs are going to come back from the first task, but for each downstream task run we want the name of the URL in the UI. FWIW, the named task run seems promising but I wonder if we can use your idea to do it with `Parameter`s.Anna Geller
from prefect import Flow, unmapped
from prefect.tasks.prefect import create_flow_run, wait_for_flow_run
from prefect.executors import LocalDaskExecutor
with Flow("MasterFlow_Mapped", executor=LocalDaskExecutor()) as flow:
returned_urls = get_urls()
mapped_flow_run_ids = create_flow_run.map(
parameters=returned_urls,
flow_name=unmapped("flow_name"),
project_name=unmapped("project"),
)
Ryan
01/10/2022, 1:54 PMBring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.
Powered by