Thread
#prefect-community
    a

    Adam Everington

    10 months ago
    General Q: What's the best way of dealing with flows with python package dependencies? For example, I use PySFTP, PyODBC, SqlAlchemy and pandas. So when I deploy this and it runs on an azure vm with none of these dependencies installed.. how best to deal with it in terms of... which storage? which agent? Do I package it up as a docker image as per @Kevin Kho’s excellent article here: https://medium.com/the-prefect-blog/the-simple-guide-to-productionizing-data-workflows-with-docker-31a5aae67c0a Looking for general advice / see what you guys do
    Anna Geller

    Anna Geller

    10 months ago
    @Adam Everington packaging your dependencies into a Docker image is a great idea! The article above shows how you can do that. In general, the agents match with run configurations e.g. DockerRun with Docker agent, KubernetesRun with Kubernetes agent, etc. And when it comes to storage, you can either package your flow together with your dependencies using Docker storage: https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/docker_pickle_kubernetes_run_custom_ecr_image.py Or you can choose to package your dependencies separately from the flow storage - this would mean that you reference your custom image with dependencies on the run configuration, and for the flow storage you may leverage one of Git storage classes (e.g. GitHub storage) or one of cloud storage classes (e.g. S3). Here is an example that shows this pattern: https://github.com/anna-geller/packaging-prefect-flows/blob/master/flows/github_kubernetes_run_custom_ecr_image.py Here you can find more examples incl. an example Dockerfile and commands to build a custom image https://github.com/anna-geller/packaging-prefect-flows
    a

    Adam Everington

    10 months ago
    @Anna Geller your github is becoming a little bible for me, thanks for your help & contributions
    Also, are you on European time or just up SUPER late?
    Anna Geller

    Anna Geller

    10 months ago
    correct, I’m working from Berlin - Germany 🙂
    a

    Adam Everington

    10 months ago
    Germany is absolutely beautiful!
    Anna Geller

    Anna Geller

    10 months ago
    And you’re from UK?
    a

    Adam Everington

    10 months ago
    I am, the less said about that the better...