One of the tasks in a flow I m working on is to download a C

Question

One of the tasks in a flow I m working on is to download a CSV file with Selenium and use the file for further processing It works well in the local environment but when we deploy to Prefect Cloud and try to access the file it does not exist Selenium doesn t give an error so I assumed the file was downloaded It seems Prefect Cloud uses the ` tmp ` directory as its working directory Does this directory delete files that enter there or the file never entered the directory I would appreciate any help

Ifeoluwa Daranijo · Answer

I am with < Toluwani Oludare> on this Please < Jeff Hale> when you get this please assist Our active deployment depends on this and it keeps failing Ideally we will like to run the flow in the same local storage where the code lives and override prefect config to download it to the ` tmp tmp prefect` before running if that is possible Thanks

Ifeoluwa Daranijo · Answer

To add more info The deployment runs on an ubuntu server It is deployed using Python prefect deployment method

Jeff Hale · Answer

You can use and absolute path when you specify where you want your created files to be stored on your infrastructure

Ifeoluwa Daranijo · Answer

The problem with that is I don t have control over that as Selenium downloads to the working directory where the script is executed

Jeff Hale · Answer

The files are downloaded to the temporary directory and then that directory is removed when the flow completes You probably could enable result persistence and save your files that way Check out the docs concept section on results

Ifeoluwa Daranijo · Answer

Let me check that out but I doubt that is the problem I further debugged the flow by adding a delay of 10 minutes and downloading the file twice using Selenium but it didn t show in the ` tmp tmp prefect` folder Lastly the file does not live outside the flow there is a command to delete the file after loading in pandas in the same flow I am guessing there is no running from using the ` tmp tmp prefect` as the location where the prefect flow is run correct

Ifeoluwa Daranijo · Answer

Here is what the output log looks like This is in a flow and the output log is from a task The task is the one downloading and loading the csv file The error says it couldn t read the file

Jeff Hale · Answer

It looks like you can specify the folder you want selenium to download to <https www browserstack com guide download file using selenium python>

Jeff Hale · Answer

Found what looks like a way to set the temp directory path <https prefect community slack com archives CL09KU1K7 p1677614773173219 thread ts=1673458911 027309 amp cid=CL09KU1K7>

Jeff Hale · Answer

I m not getting the TMPDIR environment variable to work and don t see where that would override the location in the source code Will enquire further thinking face

Jeff Hale · Answer

< Ifeoluwa Daranijo> got an option to set the working directory star Create a Process Block and set the Working Directory to whatever directory you like Specify that Process block in your deployment

Ifeoluwa Daranijo · Answer

Hey Jeff I had a break and just got back Thanks for your suggestions I will try them out now sunglasses

Ifeoluwa Daranijo · Answer

Hey < Jeff Hale> the process block works fine < Toluwani Oludare> was able to set it up using prefect python process config

Toluwani Oludare · Answer

Thanks < Jeff Hale> for the assistance heart

Jeff Hale · Answer

Great Sorry it slipped my mind at first