https://prefect.io logo
Title
r

Riccardo Tesselli

07/28/2022, 2:25 PM
this may be a stupid question, but I’ve tried to build a deployment with:
prefect deployment build flows/hello_world.py:hello_world --name "Hello world dev" --tag kubernetes --tag dev --infra kubernetes-job --infra-block base-dev
and I got this:
File "/Users/tessellir/Projects/il-da-orchestrator/.venv/lib/python3.8/site-packages/prefect/blocks/core.py", line 501, in load
    block_type_slug, block_document_name = name.split("/", 1)
ValueError: not enough values to unpack (expected 2, got 1)
An exception occurred.
I’ve ofc created a block in Prefect cloud of type KubernetesJob called base-dev, what am I missing?
1
also, in the previous iterations I quite liked the fact to be able to create a deployment via
Deployment.create()
but I saw that is dismissed, any chance to have it back in the upcoming versions?
1
m

Mathijs Carlu

07/28/2022, 2:35 PM
-ib kubernetes-job/base-dev works for me, but it should indeed work as you tried to build it
:gratitude-thank-you: 1
👍 1
r

Riccardo Tesselli

07/28/2022, 2:36 PM
it did the trick, thanks!
👍 1
couldn’t it be enough just to define
--infra-block
and infer
--infra
from the first part before the
/
?
m

Mathijs Carlu

07/28/2022, 2:51 PM
I assume you mean "infer 'kubernetes-job' from 'base-dev' ", right? If the name of an infra-block is unique across all infrastructure type, yes. But I don't know whether that's a constraint right now
r

Riccardo Tesselli

07/28/2022, 3:23 PM
now I’m trying adding a local storage to the deployment with:
prefect deployment build flows/hello_world.py:hello_world --name "Hello world dev" --tag kubernetes --tag dev --infra kubernetes-job --infra-block kubernetes-job/base-dev --storage-block local-file-system/runtime-folder
and I have the block in the image on cloud, but I get this
File "/Users/tessellir/Projects/il-da-orchestrator/.venv/lib/python3.8/site-packages/prefect/filesystems.py", line 114, in put_directory
    shutil.copytree(from_path, local_path, dirs_exist_ok=True)
NameError: name 'from_path' is not defined
An exception occurred.
m

Mathijs Carlu

07/28/2022, 3:35 PM
Local storage can't be used when using KubernetesJob infrastructure, according to the docs: https://orion-docs.prefect.io/concepts/infrastructure/#kubernetes-flow-runner (link is okay, title has yet to be modified). Actually, I think you're trying something similar to what I am. You also want to store your flow inside a docker image, that is then executed as a kubernetes job?
a

Andreas Nigg

07/28/2022, 8:15 PM
Hey, I succeeded in having prefect 2.0.0 run as KubernetesJob. Maybe these steps help, if not, sorry for hijacking your thread :-) 1. Create a Docker image containing all my dependencies 2. Create a GCS Storage Block in prefect UI called my_gcs_block 3. Deploy an Agent on kubernetes 4. Create a depoyment with prefect deploymemt build <pathtoflow.py:flowname> --name myname_deployment -i kubernetes-job -sb gcs/my-gcs-block 5. In the resulting deployment.yaml, I manually adjusted the Kubernetes Job settings. Specifically, env, image ans service_account_name 6. prefect deploymemt apply... 7. prefect deploymemt run (note that execute does not work, only run) I think its good to define the KubernetesJob in the deployment.yaml rather than in the UI, because it helps me to better integrate it with our infrastructure as code strategy. Not sure if this is helpful - if you want me to post my deployment configs or more details about some individual step, let me know
l

Lucien Fregosi

07/29/2022, 12:13 PM
Hi @Andreas Nigg it helps lot thank you. I’m now wondering about the best way to edit env, image and service_account_name programmatically for automated deployment. Currently using sed but a it’s a bit hacky
a

Andreas Nigg

07/29/2022, 12:19 PM
For me it's also sed at the moment. Maybe for automated deployment, first defining a KubernetesJob block in the prefect UI (or with the python client) and then using this precreated block with "prefect deploymemt build -ib <block-name>" would work well in such a scenario?
l

Lucien Fregosi

07/29/2022, 12:46 PM
Hey you idea si great I haven’t been discovered Kubernetes job as a block before your message ! Thanks !
r

Riccardo Tesselli

07/30/2022, 9:37 PM
@Mathijs Carlu that is exactly what I want to do sir, have you managed to find a workaround?
m

Mathijs Carlu

08/01/2022, 7:18 AM
@Riccardo Tesselli yes, you can find it over here
🙌 1