https://prefect.io logo
Title
c

Cole Murray

05/07/2022, 3:53 AM
Hey Prefect community, I'm working on deploying an instance of an Orion server to ECS using AWS CDK. One issue I've run into is how the database url is being specified via env vars. In ECS, there is a distinction between environment_vars and secrets and, to my knowledge, they cannot be concatenated. This presents a dilemma as you know must: ā€¢ Have the database password in the clear as an env var. ā—¦ This also opens up an issue, where if the password is updated, it is not reflected until the next deployment. ā€¢ resolve the secret key, concat with the whole URI, put into SSM
orionServer.addContainer('ServiceContainer', {
            essential: true,
            image: ContainerImage.fromDockerImageAsset(serviceImage),
            logging: new AwsLogDriver({
                streamPrefix: 'orion-server'
            }),
            portMappings: [{
                containerPort: this.props.orionEnvVars?.port || Number(this.CONTAINER_PORT)
            }],
            environment: {
                PREFECT_ORION_API_HOST: this.props.orionEnvVars?.host || '0.0.0.0',
                PREFECT_ORION_API_PORT: this.props.orionEnvVars?.port?.toString() || this.CONTAINER_PORT,
                PREFECT_ORION_DATABASE_CONNECTION_URL: `postgres+asyncpg:///${this.props.username}:${ecsSecret.fromSecretsManager(this.props.dbPassword)}@${this.props.databaseHost}/orion`

            },
            secrets: {

            },
            memoryLimitMiB: 300,
        });
We can overcome this by altering the settings provided, and fetching the username and password separately from the host to build the DB URI, perhaps as a second set of options. Has anyone found an alternate way to concat the strings together? If not, are we open to adding additional options to distinguish DB_PASSWORD to be injected as secrets?
Something like this should work:
PREFECT_ORION_DATABASE_PASSWORD = Setting(
    str,
    default="",
    description=textwrap.dedent(
        """
        Password of the orion database. Intended to be used with string templating in database connection url.

        Usage: postgres+asyncpg:///postgres:${PREFECT_ORION_DATABASE_PASSWORD}@localhost/orion
        """
    )
)

PREFECT_ORION_DATABASE_CONNECTION_URL = Setting(
    str,
    default="sqlite+aiosqlite:////${PREFECT_HOME}/orion.db",
    description=textwrap.dedent(
        f"""
        A database connection URL in a SQLAlchemy-compatible
        format. Orion currently supports SQLite and Postgres. Note that all
        Orion engines must use an async driver - for SQLite, use
        `sqlite+aiosqlite` and for Postgres use `postgresql+asyncpg`.

        SQLite in-memory databases can be used by providing the url
        `sqlite+aiosqlite:///file::memory:?cache=shared&uri=true&check_same_thread=false`,
        which will allow the database to be accessed by multiple threads. Note
        that in-memory databases can not be accessed from multiple processes and
        should only be used for simple tests.

        Defaults to a sqlite database stored in the Prefect home directory.
        """
    ),
    value_callback=template_with_settings(PREFECT_HOME, PREFECT_ORION_DATABASE_PASSWORD),
)
a

Anna Geller

05/07/2022, 11:27 AM
Could you open a GitHub issue for it? This looks like a feature request rather than a support question. Feature requests and bug reports are easier to communicate through GitHub as this way we can link issues with PRs and roadmap items
šŸ‘ 1
c

Cole Murray

05/08/2022, 12:34 AM