https://prefect.io logo
f

Fernando Silveira

11/22/2022, 3:39 AM
Hi folks I'm trying to setup my Prefect v2 flows to run in `KubernetesJob`'s but I'm having trouble figuring out the best way to setup my storage block. In my deployments, I create a docker image with all my flow code and corresponding dependencies. I wish I could use something like the
DockerContainer
storage block to my deployment indicating that it should just pull my docker image from AWS ECR and run the flow. However, as far as I understand, the
DockerContainer
block is also an infrastructure block and is meant to run the image against docker directly on a machine - i.e., NOT in a
KubernetesJob
. This means that so far, I've been relegated to using the
S3
storage block which pushes my flow code to S3, only to pull it back to my docker image at flow run time (when the image already contains all the code it needs to run). This feels wrong so I'm sure I must be missing something here. How do I configure
KubernetesJob
to just pull a docker image and not rely on any additional storage block like S3? PS: so far, I'm really enjoying how quickly I'm getting up to speed on prefect v2. Despite of kinks like this, I'm really enjoying the development experience.
1
z

Zanie

11/22/2022, 4:08 AM
Hi! Just set an
image
on your Kubernetes Job — although for ECR it’s a little tricky to authenticate.
f

Fernando Silveira

11/22/2022, 4:10 AM
I am doing that but do you mean that I don't need to specify the flow storage for that deployment?
z

Zanie

11/22/2022, 4:10 AM
See https://github.com/PrefectHQ/prefect-aws/pull/68 for WIP on a block for ECR authentication
Ah, I see. Yeah you’d set it to the path to the flow in the image.
f

Fernando Silveira

11/22/2022, 4:11 AM
I have the ECR auth down - I am running the flow in the correct image pulled from ECR. I see
so local storage
which in the k8s job will mean local inside the pod container
Thanks @Zanie - I'll try that!
z

Zanie

11/22/2022, 4:12 AM
Yeah, I think you might need to specify to skip upload too.
f

Fernando Silveira

11/22/2022, 4:12 AM
oh, how do I skip upload?
is that a parameter somewhere?
z

Zanie

11/22/2022, 4:13 AM
I’m not sure where our latest demo of that is tbh, I can find out tomorrow.
f

Fernando Silveira

11/22/2022, 4:13 AM
no worries, thanks a lot
have a good night!
z

Zanie

11/22/2022, 4:13 AM
There’s a
--skip-upload
flag on the
prefect deployment build
command
👍 1
Thanks!
f

Fernando Silveira

11/22/2022, 6:04 AM
Just to confirm - this worked! I was able to bypass S3 storage and just use my docker image
🙌 2