Ricardo Garcia Silva
08/07/2025, 5:53 PMprocess
and since the cluster cannot access the Internet, I'm also storing flows locally on each node. So I have setup a couple of nodes, each running prefect worker
with a work pool of type process
.
I have successfully came up with an ansible playbook which deploys my sample flow to the cluster. However, I would like to be able to use an isolated environment for each flow - how can I achieve this when using the process
work pool type?
I have already created a venv for my flow and would now like to configure the prefect deployment with instructions to use it. This is my prefect.yaml
file:
deployments:
- name: "my_workflow"
entrypoint: "main.py:my_workflow"
work_pool:
name: green
work_queue_name: low
job_variables:
command: |
echo "About to activate" && . /opt/prefect/flows/demo1/activate.sh && /opt/prefect/flows/demo1/env/bin/python -m prefect.engine
working_dir: "/opt/prefect/flows/demo1"
unfortunately this does not seem to work - the flow is deployed successfully and I can trigger workflow runs just fine. However they do not use the custom environment, but rather run with the same environment being used by the prefect worker process.
I was hoping to be able to override the python env with that job_variables.command
property of the deployment, but it seems to be silently ignored. Is there some other way to do this, maybe using env variables?
Thanks for helping out!