Thread
#prefect-community
    v

    Valentin Willscher

    1 year ago
    We are looking into how to test our flows and even the whole system (think staging vs. production, similar to here: https://prefect-community.slack.com/archives/CL09KU1K7/p1617753993118300) Is there someone who uses physical separation / different machines or accounts for prefect to have isolation between a staging/test environment and production?
    Kevin Kho

    Kevin Kho

    1 year ago
    Hey @Valentin Willscher! Did you see this recent thread ? It may be a good starting point.
    v

    Valentin Willscher

    1 year ago
    Thank you @Kevin Kho! But I think my use case is a bit different, because I don’t want to have 2 (or 3) identical flows that run in multiple environments. My goal is to have 2 flows that run in different environments and can vary from each other
    E.g. imagine I use task/flow tagging to configure how many tasks can run at one time. If I change this configuration, I want to test my changes in a test environment and validate the result, before I make changes to the production environment
    Kevin Kho

    Kevin Kho

    1 year ago
    It sounds like the separate projects in the thread you linked should be good then right?
    v

    Valentin Willscher

    1 year ago
    Please correct me if I’m wrong, but is task tagging not a global property?
    Kevin Kho

    Kevin Kho

    1 year ago
    You’re right it is. Sorry I’m not quite understanding the need. Is it that you want to use tagging to limit the runs so that a Flow will only run in test, and then if it succeeds you will register and run in prod? I am a bit confused why you would need the tagging since you can test the Flow in an environment without the limiting?
    v

    Valentin Willscher

    1 year ago
    Sometimes it makes a big difference how many task run at a time. It can improve performance but also reduce performance or even cause an outage by overloading other systems
    In my company, we have a production environment on AWS - and a test environment which is on a separate AWS account and mirrors the production environment
    Therefore we can do load testing and other tests on the test environment to understand the effects, without impacting or endangering the production environment
    I’m looking for a solution of how to use prefect so that we can do the same thing: have one flow that we use in both test and production environments. Then, when we want to make a change, we would first make the change in the test environment and test it, then make the change in the production environment
    Task tagging is a good example - I would like to change the concurrency from say 2 to 4 and see how the change impacts other systems in the test environment. If all looks good, I would make the same change in the production environment
    Does the explanation make sense?
    Kevin Kho

    Kevin Kho

    1 year ago
    Yes I understand now. Thanks for clarifying!
    I will ask the team and get back to you tom.
    v

    Valentin Willscher

    1 year ago
    Thank you so much! 🙇
    Kevin Kho

    Kevin Kho

    1 year ago
    Hi @Valentin Willscher! The recommendation here is to use task labels like
    aws-prod
    and
    aws-dev
    . Also labels for your agent like
    dev-cluster
    and
    prod-cluster
    to achieve that separation. This way, editing the task concurrency of the
    aws-dev
    label will be detached from the setting on
    aws-prod
    v

    Valentin Willscher

    1 year ago
    Hi @Kevin Kho thank you for the quick feedback!
    I think that indeed sounds like a possible way of designing things out.
    One thing that concerns me a little bit about this prefixing style of architecture/design (and in fact, we are already using it in some places for prefect already) is that it adds quite a bit complexity both in the CICD and the code.
    Is there a plan for prefect to offer first-class support of a test environment, so that all this prefixing would essentially not be needed anymore? Or even just a pricing plan that includes having a non productive “test” environment for which one does not have to pay the full price?
    Nonetheless, thank you for your feedback, I appreciate that a lot 🙂
    Kevin Kho

    Kevin Kho

    1 year ago
    So when I worked previously as a data scientist, mirrored resources across envs were prefixed with standard naming and it worked out great for our use case. What complexity do you encounter in the CICD pipeline? No plan at the moment for test env because to be honest, most of our users use the prefix scheme. We definitely listen to feedback though and I’ll take note of this to bring up 🙂. Would still like to hear more details on the pain points it brings
    v

    Valentin Willscher

    1 year ago
    Hi Kevin, sorry for coming back a bit late, I had to gather my thoughts
    I’m coming from a more software-engineering background. So my view leans more towards how enterprises and bigger teams can use prefect on scale - but these are also well paying customers, aren’t they? 😃 So I’ve written down of my feedback that you can share with the prefect team regarding using prefixing to make the distinction between test and production environments; I hope it can help you guys to understand the view of me/us a bit better. From our perspective, the challenges are: 1) One must now in advance what entities exist and what makes them unique. For example, are tasks unique by their name or is it possible to have project-test/task1 and project-production/task1? Same for everything else (secrets, flows, tags, ...). 2) If you get it wrong or you start with a production environment only and want to add a test environment later, you are now in the situation that you have to refactor things after the fact, or you will have
    flow1
    and
    flow1-test
    . Even worse so if you have flows that already have
    test
    in their name but are production flows. They would be renamed to
    flow1-test-test
    which can be quite confusing. 3) Code quality is higher, because instead of just configuring in CICD “deploy this code to machine X or Y” I know have to do use prefixes everywhere in my code, such as
    get_secret(f”{environment}-secretX”)
    . 4) Testing anything that goes beyond things that can be prefixed is not possible. Imagine I want to give someone training of having administrative permissions on prefect, before I allow them to actually do that in the production environment. There is no way for me to do that, because I cannot give them administrative permissions for only things that are prefixed with “test”. Compliance policies might actually make it impossible to use prefect for these reasons. 5) Last but not least: standardization. The perception that people will have of Prefect will also depend on how easy it is to use it over time. If every company has their own custom way of doing testing, then this will hurt people’s ability to switch companies/jobs and get started quickly, because each time they would have to learn the company’s specific way of doing things. To some degree this is expected, but something like “test environment” is a solved problem in the industry, so we can avoid a lot of friction/overhead here. For that reason, when you look at most commercial software products, they almost always allow for a distinction between environments. I would love to see prefect adopt both permissions/role management as well as some more streamligned approach to testing (however it looks like). The potential is huge!
    Kevin Kho

    Kevin Kho

    1 year ago
    @Valentin Willscher thanks a lot for writing the very detailed feedback. This certainly helps me elevate it to the team.
    The initial feedback of the team is to have two tenants. For enterprises, this is a feature of the license. This would provide the separation. The feedback will definitely be considered and even if what we come up with does not exactly match what you described, we will incorporate it in shaping what that looks like.
    v

    Valentin Willscher

    1 year ago
    Thank you Kevin! I didn’t know that the enterprise license has this feature already - this is very good to know. I think that makes most of my feedback irrelevant then 😅
    I’ll have a look into our licensing then
    Kevin Kho

    Kevin Kho

    1 year ago
    A lot of your feedback is still relevant (number 1 particularly). I honestly don’t know what that multiple tenant feature looks like. Would you like to be put in touch with the sales folks to find out?
    Or do you have someone you work with already?
    v

    Valentin Willscher

    1 year ago
    Yes, we have already made an appointment to talk about it 👍
    Looks like we are still on an outdated license actually 🙂
    Thank you for the great support!