@Marvin Is it possible to manually set a working directory when deploying a flow to a work pool?

It seems that the direct manipulation of the working directory for a flow being executed in a work pool is not detailed in the standard documentation, at least based on these excerpts. Nonetheless, work pool behavior can be controlled to some degree, such as the number of concurrent runs allowed, but specifics on setting a working directory are not mentioned in the provided information. If you wish to specify the working directory for code executing in a Prefect flow, you might consider setting the working directory within the flow's tasks themselves using Python's

os

module to change directories, e.g.,

os.chdir('path/to/working/directory')

. It would be beneficial to search GitHub issues for related discussions or possible enhancements, as this feature or its workaround might be a topic within the community. Would you like me to search GitHub issues for you?

@Marvin Yes, please

I've found a few GitHub issues that may offer insight into the problem you're experiencing: 1. Duplicate Task runs in Cloud Run Push Work Pool (#11299): This issue highlights that tasks of a deployment were running multiple times unintentionally. The user has set up a work pool and provided a YAML configuration. Interestingly, this YAML includes a step to set the working directory (

set_working_directory

) for the flow. 2. Encountered error while running prefect.deployments.steps.set_working_directory - FileNotFoundError (#10285): A user encountered an error related to setting the working directory during flow deployment. It seems that Prefect attempts to set a working directory based on the local directory used when creating the deployment, which may not exist in the execution environment. 3. Cloning Error during pull step in deployment only after first run (#10420): The issue here is with the cloning process during the pull step of a deployment, but it's interesting as it includes the use of

set_working_directory

in the prefect.yaml. It seems that Prefect supports setting the working directory as part of the deployment process. Based on this information, it appears that Prefect does allow for setting a working directory as part of the flow deployment process through configuration settings in the prefect.yaml file. The

set_working_directory

step is a clue that you can, in fact, specify the working directory, but this functionality might be dependent upon how your Prefect environment and infrastructure are set up. If you are encountering issues or need further details on this functionality, it might help to examine the GitHub issues that I found or consult the Prefect community or documentation for updated best practices.

there are a couple ways to do it, you can do it in a pull step with prefect.deployments.steps.set_working_directory or set it on your work pool (if a process work pool) or yeah like manually os.chdir around as needed (but wouldn’t do that unless you really need to)

I’m trying to migrate from using ECS Agents to a Kubernetes work pool. Basically the flows themselves exist on my image in

/opt/prefect/flows

but Prefect appears to be looking for them in

/opt/prefect

, eg;

FileNotFoundError: [Errno 2] No such file or directory: '/opt/prefect/cowsay_example.py'

Any recommendations?

how’re you making your images

We create them manually as part of our CI/CD on merge. I know that the new

flows.deploy()

paradigm has tools for building right then-and-there but I’d like to keep our existing CI/CD structure if possible

cool, i myself like my image builds outside of prefect often as well. i like cli deploys as well personally i dont like baking source into images when possible, i like having my 3rd party deps in an image i stick on my k8s work pool (or override for that specific deployment in the prefect.yaml) and then use git_clone pull step to pull in the src, which ime changes a lot more than your images

Yeah, we had some bad experiences using remote storage back on Prefect 1 so with the Prefect 2 migration we moved to baking everything into the image. Might have been a poor long term choice 😅 Can you demonstrate how to use a pull step if I’m not actually pulling from a remote?

sorry on train @Marvin can you find the set_working_directory example in docs?

It looks like we hit a bit of a snag; the document returned doesn’t contain the specific example usage of

set_working_directory

that you are looking for. However, it provided details about changing workspaces and configuring Prefect API settings in general. You might want to try looking for the example directly in the Prefect documentation at docs.prefect.io or by checking the Prefect GitHub repository for code snippets that might reference

set_working_directory

. If you have the ability to open a web page, searching the documentation directly might be your fastest route to an answer.

gah

I should be able to dig around and find it- thanks for your help as always Nate!

👍

So I wasn’t able to find any examples of

set_working_directory

. One thing that has confused me a bit in learning these new deployment structures is what exactly the

flow.deploy()

equivalent of the

yaml

`build`/`pull` would be- I haven’t been able to find any examples of what those look like programmatically. I was able to successfully run a flow by just manually moving the files from

/opt/prefect/flows

to

/opt/prefect

, so in theory my strategy should work if I can just figure out how to change the working directory. Unfortunately _`PREFECT_LOCAL_STORAGE_PATH`_ doesnt seem to work 😆

Yeah that’s exactly right- except we’re using

flow.deploy()

instead of defining it in yaml