https://prefect.io logo
Title
r

Ross Leung

03/29/2023, 7:29 PM
Hi, quick question. I have a simple write to file in one of my tasks:
def task_1():
    
    from prefect.filesystems import LocalFileSystem
    fs = LocalFileSystem(basepath="./")
    with open("task_out.txt","w") as f:
        f.writelines('Hi')
        fs.write_path(path="task_out.txt",content=f)
    
    return "task_out.txt"
Basically, I want to write to a file called
task_out.txt
. I tried it without using the
prefect LocalFileSystem
and simply used the
with open
file context, but the file didn’t appear.
j

Jarvis Stubblefield

03/30/2023, 4:23 PM
Ross, in at least one of my flows I write output to a file to be sent to a customer… how are you deploying/running this flow? I’m asking how you’re running it because most flows are executed in a temporary path which would mean your file would be written to this temporary location… if you want to control where your flow runs from you’ll need to specify that in deployment…
Here’s the task I use to write the file…
@task
def write_log_report_to_file(filename: str, fieldnames: List[str], logs_report: List[dict]) -> None:
    with tmpchdir(flow_files_path):
        with open(filename, 'w') as csvfile:
            # This is for MS Excel
            csvfile.write(u'\ufeff')
            writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
            writer.writeheader()
            for report in logs_report:
                writer.writerow(report)
This is where I create the
flow_files_path
from tenzinga_django_setup import prepare_django_and_paths
tenzinga_path, ppower_path = prepare_django_and_paths()
flow_files_path = tenzinga_path.joinpath("web-write").joinpath("flow-files")
Also, for the
tmpchdir
functionality I pulled that directly from Prefect…
from prefect.utilities.filesystem import tmpchdir
I hope this is helpful to you in your journey!
r

Ross Leung

03/31/2023, 12:28 AM
ahh yes, thank you for the tips. I confirmed it by printing out the
os.getcwd()
and indeed my flow is ran in a tmp folder that disappears. I also confirmed that if I put a full local directory path instead of a relative path, the output goes to the right location too. Thanks!
👍 1