Small question (or 2) about prefect 2.0 Trying to...
# prefect-community
n
Small question (or 2) about prefect 2.0 Trying to build a CICD solution with prefect 2.0. There are a few things I’m still struggling with; • What is the advantage of storing my flow on another source than Git? Using GIT gives me a single source of truth inline with the standards in my organization. • Where does my flow store its cache while running? I want to be able to restart a task when my flow crashed, while running on K8s. Am I correct when assuming that I need to configure External Storage with put-directory and write-path for this? • The MINIO example on the website is trying to connect to an AWS endpoint, probably due to s3fs. Can I just follow the example on https://docs.prefect.io/concepts/blocks/, create my own PyMinio based block and use that? Many thanks in advance
1
a
#1 Git is way more than just storage, typically engineers use it for version control and collaboration and then CI/CD deploys the application upon merge to main -- a common scenario in Prefect is that your flow along with your custom modules gets deployed to S3 and your dependencies are built into Docker image and pushed to a registry; or maybe you are leveraging both as part of flow code stored in the Docker image as shown here - we do have plans to support Git as storage but if you want to read more on that check this Discourse page #2 no caching, Prefect always pulls flow code from storage at runtime due to the hybrid model, only container image can be cached using image pull policies
n
Hi Anna, thnxs for the quick reply #1 Fair point. I will have to look into this how this works in best practice with Prefect. Will look at the link you send me. #2 So if I understand correctly, when a task fails, the flow should run as a whole again. There is no option to start te flow from the task that failed when running in a Docker / k8s environment.
a
correct, unless you leverage automatic retries which are configurable both on a task and a flow level in Prefect 2
🙌 1