Thread
#prefect-server
    Tony Liberato

    Tony Liberato

    6 months ago
    Good morning, Prefectionists! We are sailing. I have successfully deployed two very simple Workflows about a month ago and everything is working beautifully. I'm a DBA for a small non-profit and we handle a ton of data. We decided to implement prefect so I can automate many of the data importing and ETL tasks I perform manually. I wanted to share a couple of things Prefect taught me. 1:Register the flow on the agent its intended to run on. for some reason this resolved all the import statements correctly. 2:Translating your workflow into prefect code helps to refactor the steps of the flow. I would do these in SSIS, and SSIS made the "theory" of the flow easy, but it didn't truly represent the logic of the flow. In doing so, I developed a much more concise workflow, which had a few more tasks. Let this happen: its a good thing. 3: Python is amazing. And since Prefect doesn't get in the way of that, prefect is amazing. It quite literally can do anything. 4: Docker flows: This can get messy. I found about 52 containers that were left over from all my flows being run. This is a really intelligent solution regardless, it just leave a trail. I'm sure there is a setting to automatically remove these that I just haven't found. Don't be alarmed if you have clutter in your docker instance. It well worth it. 5: When deciding to implement a flow in Prefect, you are forced to design the whole thing. No more "Oh, we'll deal with that when we get there..." This is a steep hill to climb, but you once you do, its a never-ending coast down the other side. This forces questions to the beginning of the process and helps get solid answers for them right away. Its possible that a workflow may not be a good candidate for Prefect, or maybe parts of the workflows are candidates. You actually save a lot of time designing end to end. Thanks for reading.
    Anna Geller

    Anna Geller

    6 months ago
    We usually call Prefect users “Prefectionists”! 😄 Thank you so much for sharing your experience! 👏 #1 - nice that your imports work as intended - if you face any issues with packaging your code dependencies, let us know, we have some resources that can help with that #2 - I definitely empathize with you since I also used SSIS, Pentaho Data Integrations and similar drag-and-drop ETL madness and I can attest that it’s not a good experience, creates a ton of tech debt, is hard to scale and makes it really hard to implement best engineering practices #3 That’s true! 💪 #4 - You could use a ShellTask in a flow running e.g. every day/week and running the command
    docker container prune
    This allows you to remove stopped containers and reclaim the disk space. You may even do:
    docker system prune
    #5 - I agree, Prefect allows you to build workflows in a very modular and flexible way. Again, thanks for sharing! 👏
    Kevin Kho

    Kevin Kho

    6 months ago
    standing ovation
    Tony Liberato

    Tony Liberato

    6 months ago
    Thanks, Anna. I will definitely use your suggestions.
    <bow>