Hi, I am honestly not really happy with the direct...
# prefect-community
Hi, I am honestly not really happy with the direction Prefect 2.0 is going... Prefect 1.0 was decent, but the current setup really confuses me... I still did not figure out what the best way would be to schedule existing Kubernetes Cronjobs? I know there is prefect-kubernetes, but according to the Medium post you need an external storage? Why not just the Docker container? I mean all files are baked in there, including persistent volumes and everything... Hence my question: What is the best way to schedule existing Kubernetes Jobs/Cronjobs using Prefect, with the background info that the containers use persistent volumes and env vars? example:
Copy code
# apiVersion: v1
# kind: PersistentVolumeClaim
# metadata:
#   name: composer-bigtechmomentum-v1-pvc
# spec:
#   accessModes:
#     - ReadWriteOnce
#   resources:
#     requests:
#       storage: 1Mi
# ---
apiVersion: batch/v1
kind: CronJob
  name: composer-bigtechmomentum-v1
  schedule: "42 3 * * *"
  suspend: false
  concurrencyPolicy: Forbid
          - name: composer-bigtechmomentum-v1
            image: guestros/xxxxxxxxxx:latest
            imagePullPolicy: Always
          #   volumeMounts:
          #     - mountPath: /app/persistent/
          #       name: composer-bigtechmomentum-v1-pv
tagging @jawnsy because i saw his reply in a similar post
hey @Justin, you can tweak the manifest in anyway you wish using infra_override patches
out of interest why would you want to use the k8s scheduler rather than let prefect orchestrate?
no no, i would be fine with prefect, i just didn't see a straightforward way to convert my "old" cronjobs to kubernetes... some example would be nice, because as I saw it the medium post does not apply to this case, because it uses an external s3 storage
btw if there would be a script to paste in existing k8s cronjob yaml's, and they "convert" to the prefect manifest automatically that would make it easier for a lot of businesses to switch...
I'm sure they will get to a few of those things as 2.0 matures
scheduling is easy - just pass in a CronSchedule to the deployment e.g
hm, thanks already for your help, but I still don't see how it can be applied to my use case... Because as I said, I do not want to store the flow.py seperately, I just want to run the docker image like a k8s cronjob would do, or even as simple as "docker run guestros/xxxx:latest" As far as I can see, you always need to provide a seperate flow.py, which would require for me to rewrite all docker images, store the flow.py's seperately (in s3), instead of just running the docker image like k8s and docker would do...
I think you can specify storage as none then provide your own docker image - as long as the path is set correctly it should pick it up
👍 1
I've looked at the source code to grok that
alright nice? but i would still need to write a "flow.py" to trigger the start of the image? that sounds like an unnecessary step? wouldn't it be more logical to create a deployment from this k8s-job block?
sorry not sure - I use s3 as storage and keep the flows separate to prevent mass container rebuilds
🙌 1
alright, thx anyways! helped a lot 🙂
i basically want a "KubernetesJobOperator" like Airflow is providing
Copy code
job_task = KubernetesJobOperator(
    command=["bash", "-c", 'echo "all ok"'],
https://github.com/LamaAni/KubernetesJobOperator I just think that Airflow is a) quite resource heavy and b) prefect has a nicer UI, so would be a shame to switch
and much cheaper compared to say aws airflow
how do you handle imports with the s3 setup? like if i have a "helper.py" which contains a "primenumbers" function? just drop it into the s3 as well?
and does it handle outputs as well? like let's say my script outputs a results.txt and it appears in the s3?
During deploy I copy a shared function folder below my flow so it gets uploaded