Hi, quick question. I have a simple write to file in one of my tasks: ```def task_1(): from...

Ross Leung

03/29/2023, 7:29 PM

Hi, quick question. I have a simple write to file in one of my tasks:

Copy code

def task_1():
    
    from prefect.filesystems import LocalFileSystem
    fs = LocalFileSystem(basepath="./")
    with open("task_out.txt","w") as f:
        f.writelines('Hi')
        fs.write_path(path="task_out.txt",content=f)
    
    return "task_out.txt"

Basically, I want to write to a file called

task_out.txt

. I tried it without using the

prefect LocalFileSystem

and simply used the

with open

file context, but the file didn’t appear.

Jarvis Stubblefield

03/30/2023, 4:23 PM

Ross, in at least one of my flows I write output to a file to be sent to a customer… how are you deploying/running this flow? I’m asking how you’re running it because most flows are executed in a temporary path which would mean your file would be written to this temporary location… if you want to control where your flow runs from you’ll need to specify that in deployment…

Jarvis Stubblefield

03/30/2023, 4:26 PM

Here’s the task I use to write the file…

Copy code

@task
def write_log_report_to_file(filename: str, fieldnames: List[str], logs_report: List[dict]) -> None:
    with tmpchdir(flow_files_path):
        with open(filename, 'w') as csvfile:
            # This is for MS Excel
            csvfile.write(u'\ufeff')
            writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
            writer.writeheader()
            for report in logs_report:
                writer.writerow(report)

Jarvis Stubblefield

03/30/2023, 4:27 PM

This is where I create the

flow_files_path

…

Copy code

from tenzinga_django_setup import prepare_django_and_paths
tenzinga_path, ppower_path = prepare_django_and_paths()
flow_files_path = tenzinga_path.joinpath("web-write").joinpath("flow-files")

Jarvis Stubblefield

03/30/2023, 4:27 PM

Also, for the

tmpchdir

functionality I pulled that directly from Prefect…

from prefect.utilities.filesystem import tmpchdir

Jarvis Stubblefield

03/30/2023, 4:28 PM

I hope this is helpful to you in your journey!

Ross Leung

03/31/2023, 12:28 AM

ahh yes, thank you for the tips. I confirmed it by printing out the

os.getcwd()

and indeed my flow is ran in a tmp folder that disappears. I also confirmed that if I put a full local directory path instead of a relative path, the output goes to the right location too. Thanks!

👍 1

4 Views

Open in Slack

Previous Next

Prefect Community

Bring your towel and join one of the fastest growing data communities. Welcome to our second-generation open source orchestration platform, a completely rethought approach to dataflow automation.