Frederick Thomas
04/30/2021, 4:01 PMdef register__flow():
with Flow("FlowName", schedule=None, ) as flow:
file_name = Parameter(name='file_name',default=None)
params = get_params()
p = get_file_blob(
file_name=f"notebooks/{file_name}",
con_string=params["uploads_blob"],
container="uploads"
)
flow.add_edge(file_name,params)
flow.add_edge(params, p)
if len(sys.argv) > 1 and sys.argv[1] == "register":
flow.register(project_name="project_x")
elif len(sys.argv) > 1 and sys.argv[1] == "test" and len(sys.argv[2]) > 1:
flow.run(parameters={'file_name':sys.argv[2]})
else:
print("Please use a quoted string of the file name...")
if __name__ == "__main__":
register__flow()
ThanksKevin Kho
Frederick Thomas
04/30/2021, 4:06 PMFrederick Thomas
04/30/2021, 4:08 PMFrederick Thomas
04/30/2021, 4:09 PMKevin Kho
Frederick Thomas
04/30/2021, 4:10 PMKevin Kho
import prefect
from prefect import Flow, task, Parameter
import sys
@task
def test(x):
logger = prefect.context.get("logger")
<http://logger.info|logger.info>(x)
return
with Flow("test") as flow:
param = Parameter("param", default= sys.argv[2])
test(param)
flow.run()
Kevin Kho
python flow.py test1 test2
Kevin Kho
Frederick Thomas
04/30/2021, 4:25 PMFrederick Thomas
04/30/2021, 4:32 PMdef register__flow():
with Flow("FlowName", schedule=None, ) as flow:
file_name = Parameter(name='file_name',default=sys.argv[2])
params = get_params()
p = get_file_blob(
file_name=f"notebooks/{file_name}",
con_string=params["uploads_blob"],
container="uploads"
)
flow.add_edge(file_name,params)
flow.add_edge(params, p)
if len(sys.argv) > 1 and sys.argv[1] == "register":
flow.register(project_name="project_x")
elif len(sys.argv) > 1 and sys.argv[1] == "test" and len(sys.argv[2]) > 1:
flow.run()
else:
print("Please use a quoted string of the file name...")
if __name__ == "__main__":
register__flow()
Kevin Kho
get_file_blob
there a task?Kevin Kho
filename = task(lambda p: f"notebooks/{p}")(file_name)
inside get_file_blob
. This is because file_name
has no value yet when you do that. Parameters get values during runtime and this string is given a value at build time. You need some helper task to generate the file name during runtime as task execution is deferred.Kevin Kho
Frederick Thomas
04/30/2021, 5:47 PMp = get_file_blob(
file_name = task(lambda p: f"notebooks/{p}")(file_name),
con_string=params["fileuploader_blob"],
container="fu-uploads"
)
We're currently using version 0.14.12, is there any documentation that is would emphasize this requirement? Otherwise, thanks!Kevin Kho
tasks
are used to defer execution. Maybe this documentation on tasks will help you