https://prefect.io logo
m

Moises Vera

02/03/2022, 11:20 PM
Hello, hope this would be the right channel for this question... • I already have my tasks working in my local machine, actually right now running my tasks manually, and I wanted to start deploying this tasks to a production env and prefect cloud • This is an ETL common process but with storing files locally to not waste too much RAM among tasks and because the requests I'm doing to extract data are really heavy • So, in the extract process, after I just finish a smaller task, I store the result in a local file • all of this is working in my local machine with no issue with flow.run() • But when I register the flow for prefect cloud the issues start ◦ right now I'm dealing with
FileNotFoundError: [Errno 2] No such file or directory: 'data/file.csv'
◦ this in prefect cloud ◦ mine is basic config. local agent in a server ◦ my code in a directory where I want to create the file with a directory ◦ the line of code is a
pathlib.mkdir
sentence I'm sure there is something I dont understand about prefect, because I cannot manipulate the local filesystem in the server with prefect cloud, any ideas?
k

Kevin Kho

02/04/2022, 12:36 AM
I think you just need to set the working directory of your Flow run like the last example here . It sounds like your Agent is running in a different directory so the Python process starts there
m

Moises Vera

02/04/2022, 12:40 AM
Genius!!!!
interesting how that piece of code fix everything
thank you Kevin !
k

Kevin Kho

02/04/2022, 12:43 AM
Of course!
m

Moises Vera

02/04/2022, 12:47 AM
can you clarify why is that important in this setup? and it's not required in my local machine ? it's not really clear for me
I appreaciate it
k

Kevin Kho

02/04/2022, 12:48 AM
Because the agent starts the Python process. Your agent is in another folder so it looks for the files in that folder but it’s not there. When you do
flow.run()
, you are running in in the right folder by default
m

Moises Vera

02/04/2022, 12:48 AM
aaaaaaah
ok ok, get it, thank you
4 Views