w

    Wesley Jin

    7 months ago
    Hey Prefect community! Iโ€™m using ECS task runs with Github storage for my flows but running into the following error when running the task:
    Failed to load and execute Flow's environment: ModuleNotFoundError("No module named 'constants'")
    Iโ€™m importing constants which will be shared across many flows: the project name, ECS cluster name, Github Token secret name from a
    constants.py
    file in the same directory. Running the flow locally with
    flow.run()
    works just fine as expected, but when running it in ECS
    constants
    is not found. How do I register other module dependencies with Github storage, if possible? Or do I have to add these constant values to each of my flow files? Thank you! Example below:
    . (repo root)
    |____flows
    | |____hello.py
    | |____constants.py
    | |______init__.py
    Anna Geller

    Anna Geller

    7 months ago
    Are you on Prefect Cloud? This way maybe you could store those constants in KV Store instead. The issue is that Github storage can only store your flow, but not the flow's dependencies and utilities such as your constants.py file. You would need to either build and install it as a package or change the strategy how you generate those constants e.g. moving to KV Store/AWS Parameter store or using Prefect Parameter tasks instead... just some ideas this thread also discussed this problem
    an alternative would be to use Docker storage to package both flow and its code dependencies into a single Docker image that you can push to ECR - this way you would have one image per flow
    w

    Wesley Jin

    7 months ago
    Yes weโ€™re on Prefect Cloud - and that makes sense. Will evaluate both those options! Thanks for the thread link and context on the dependencies being separate from the flow ๐Ÿ™‚
    Kevin Kho

    Kevin Kho

    7 months ago
    Hey Wesley, could you move the code in this thread when you get the chance so we can keep the main channel a bit more compact?