Is there anywhere I can get an example workflow, or get pointed to an example workflow which looks something like the following: • starts flow and creates a unique local directory for that run on disk • executes a task which creates a file - eg downloads a file from some URL using

urllib

, and saves it in

$RUN_LOC/my_file

• execs a task which runs a bash command - eg

wc -l $RUN_LOC/my_file > file_len

Obviously the tasks here are silly. My aim is to have an example of something which I can have a series of tasks do work on files. The task results are almost not important, the result might be nothing more than a RAN_OK/ NOT_OK. What matters to me is I can have a mechanism to operate on files, and shell out to other utils to operate on files and check their return status. I've been looking at https://docs.prefect.io/core/concepts/results.html#how-to-configure-task-result-persistence

from prefect import task, Flow
from prefect.engine.results import LocalResult


@task(result=LocalResult(location="initial_data.prefect"))
def root_task():
    return [1, 2, 3]

@task(result=LocalResult(location="{date:%A}/{task_name}.prefect"))
def downstream_task(x):
    return [i * 10 for i in x]

with Flow("local-results") as flow:
    downstream_task(root_task)

Hi @Philip MacMenamin, just want to clarify your use case. For each run, you'd like to: • create a unique directory wherever your flow is running • download something to that unique directory • execute some arbitrary shell command on the file in that directory, correct?

The uniq dir per run isn't essential, if that's very difficult. What I do need is to be able to run tasks, on files, and have the task know where that file is.

I have a potentially silly question: if the results aren't important, are you attached to using results? This almost sounds like a use case better suited to two

ShellTasks

, potentially with a Python task sandwiched in between to download your file.

No! I initially didn't think results was what I wanted to use.

Okay, awesome. This is a toy flow, so you'll need to customize it to whatever you're trying to do, but here's a super naive example:

from prefect import task, Flow
from prefect.tasks.shell import ShellTask

directory = "testing456"

mkdir_task = ShellTask()
# in your use case you'd download the file
# rather than using touch_task
touch_task = ShellTask()
with Flow("shelltaskflow") as flow:
        mkdir_task(command=f"mkdir {directory}")
        touch_task(command=f"touch {directory}/test.txt")

out = flow.run()

OK. And, if I touch nothing else, I should see

testing456

in... the ~.prefect/results dir?

Ah, not quite. So in the case of the script I sent you, you'll see

testing456

in whatever directory you ran your flow in.

ok, that's fine. Np. Again, I can set this. So you simply scope out the dir to be visible to the tasks, and everything uses that. That seems fine.

Okay, solid! Is there anything else we can help with?

ok. Yes, I think that's good.

Woohoo, glad I could help. 🎉

sounds like the next version of JQuery

As I spelled it out again, I kicked myself for not snagging that as a Github handle. 😂