st dabu
08/04/2021, 3:37 AMwith Flow("NLP") as nlp_flow:
aws s3 cp blah blah
sometool /data/a.csv
Is it possible to add the steps as cli commands instead ?Kevin Kho
from prefect.utilities.tasks import task
from prefect import task, Flow
from prefect.tasks.shell import ShellTask
@task
def files():
return [
'/opt/file1.txt',
'/opt/file2.txt',
'/opt/file3.txt',
'/opt/file4.txt',
]
@task
def format_commands(file_name):
return f'echo "{file_name}"'
rm_task = ShellTask()
with Flow('shell') as flow:
files_to_delete = files()
commands = format_commands.map(files_to_delete)
rm_task.map(command=commands)
flow.run()
st dabu
08/25/2021, 7:55 AM