https://prefect.io logo
Title
t

Tolga Karahan

02/15/2023, 10:05 AM
Hi guys. When I build and apply a deployment I'm getting error below. Storage block is not set, thus local storage is used. Infra block specifies k8s cluster and an agent works in there, so basically I want to run a flow from my local storage in k8s cluster specified by infra block. Any idea?:
1
r

redsquare

02/15/2023, 10:22 AM
local storage is not supported for Kubernetes Jobs
🙌 1
t

Tolga Karahan

02/15/2023, 1:29 PM
@redsquare thanks
Is docker storage supported for k8s jobs to pull image from a registry and run as a job?
r

redsquare

02/15/2023, 1:50 PM
Interesting - perhaps you could fool it by setting ignore storage and setting the path to the location in the docker image https://github.com/PrefectHQ/prefect/blob/main/src/prefect/deployments.py#LL156C9-L156C27
t

Tolga Karahan

02/15/2023, 2:27 PM
@redsquare so Docker storage is not supported anymore by default?
r

redsquare

02/15/2023, 2:29 PM
it is but not sure if you use your own docker image within a k8s job->pod - does it expect the storage to come remotely
t

Tolga Karahan

02/15/2023, 2:31 PM
yeah it was how we use Prefect 1
r

redsquare

02/15/2023, 2:33 PM
looks like it will be find if you set ignore_storage - will just use the path for the flow
t

Tolga Karahan

02/15/2023, 2:36 PM
okay, thanks @redsquare 🙏
r

redsquare

02/15/2023, 2:38 PM
wonder if you could set that, attach a volume by a patch in a customization for the k8s job it would work
👍 1
t

Tolga Karahan

02/16/2023, 2:29 PM
I just wonder if flow code doesn't come from remote, but embedded in builtin image. May I use this image as the image for infra block to execute flows? Thanks in advance. 🙏
r

redsquare

02/16/2023, 2:34 PM
I have not tried but see no reason for a limitation here to allow you to run a custom docker image in a k8s job and reading the flow from the image
If you set the ignore storage to true
you should be able to store the flows on a volume in k8s and have the pod mount the volume+exxecute the flow
t

Tolga Karahan

02/16/2023, 2:44 PM
I see thank you, we'll give it a try, our main issue is we have quite heavy initialization so we want to use our custom image
r

redsquare

02/16/2023, 2:49 PM
feels a pain to build and publish a new image everytime a flow changes
or you add a new one
t

Tolga Karahan

02/16/2023, 4:07 PM
yeah
so it looks like using custom image as execution environment but getting flows from some storage is better
r

redsquare

02/16/2023, 4:08 PM
100%
t

Tolga Karahan

02/16/2023, 4:09 PM
I'll try using custom image as infra block image, and also a storage maybe s3 or remote git repo
r

redsquare

02/16/2023, 4:09 PM
we use k8s with s3 storage - works great
t

Tolga Karahan

02/16/2023, 4:12 PM
your pods are also using a custom image?
or prefects own image
r

redsquare

02/16/2023, 4:13 PM
currently prefects own until we are sure about common requirements across all flows - then we will create a custom image - but only when that is clear
time for the flow to run is not uber critical for us currently - so we have lots of pip installs
lots = 5-15
t

Tolga Karahan

02/16/2023, 4:14 PM
I see, sounds good
ML dependencies are a bit problem, libraries are heavy
r

redsquare

02/16/2023, 4:15 PM
yeah, we are light on ML but use MindsDB on top of Clickhouse currently
t

Tolga Karahan

02/16/2023, 4:15 PM
oh interesting
I know it but didn't find any occasion to use 🙂
r

redsquare

02/16/2023, 4:16 PM
click or minds:)
t

Tolga Karahan

02/16/2023, 4:17 PM
MindsDB, have no idea on click 😄
r

redsquare

02/16/2023, 4:20 PM
They are both great, clickhouse especially
t

Tolga Karahan

02/16/2023, 4:22 PM
yeah, mindsdb look very interesting, I'll take a look to other