Jacqueline Riley Garrahan
03/06/2022, 6:06 AMapiVersion: batch/v1
kind: Job
metadata:
name: prefect-job
labels: {}
spec:
template:
metadata:
labels: {}
spec:
containers:
- name: flow
image: prefecthq/prefect:latest
imagePullPolicy: IfNotPresent
...
volumeMounts:
- name: filesystem-dir
mountPath: /job-files
volumes:
- name: filesystem-dir
hostPath:
# directory location on host
path: /path/to/my/dir
# this field is optional
type: Directory
I'm unable to mount the path which I believe is a function of the volume not being available on the agent. Is there a straightforward way for me to mount a volume to the agent in the helm chart deployment?Anna Geller
03/06/2022, 12:30 PMFROM prefecthq/prefect:1.0.0-python3.8
WORKDIR /opt/prefect
COPY your_dir/ /opt/prefect/your_dir/
#2 Build and push to some container registry:
docker build -t yourorg/your_image:latest .
docker push yourorg/your_image:latest
#3 Use this image in your job template:
apiVersion: batch/v1
kind: Job
metadata:
name: prefect-job
labels: {}
spec:
template:
metadata:
labels: {}
spec:
containers:
- name: flow
image: yourorg/your_image:latest
imagePullPolicy: IfNotPresent
...
Following this approach, there is no confusion where to mount a file directory (flow run pod vs agent pod) and your image can be even easier tested locally before deploying it to Kubernetes
And btw, if you're just getting started with Prefect, Prefect Cloud is much easier to use than Prefect Server since you don't have to deploy anything other than your agent (which can be started with a single command), and it's free to use for 20,000 task runs every month, no credit card required. Check out this page for more info.Jacqueline Riley Garrahan
03/06/2022, 5:18 PMAnna Geller
03/06/2022, 5:56 PMfrom prefect.run_configs import KubernetesRun
with Flow(
name="your_flow", run_config=KubernetesRun(job_template=your_job_template)
) as flow:
But what would be probably easier in such on-prem setup would be to use a Docker agent? You could then mount the volume directly on the agent and this would make it available to all flow run containers (.aws is just an example):
prefect agent docker start --label AGENT_LABEL --volume ~/.aws:/root/.aws
Jacqueline Riley Garrahan
03/06/2022, 6:55 PMdirectory not found
error on the job deploymentAnna Geller
03/06/2022, 8:23 PMJacqueline Riley Garrahan
03/07/2022, 6:12 AM