Thread
#prefect-community
    Prabin Mehta

    Prabin Mehta

    1 year ago
    Hi Everyone, I am running prefect using kubernetes Run with Dask executor. I am passing the ecr repo for image in kubernetesRun function. Some of my flows are running perfectly while some give an error for not able to get secrets . And for flows which are running they too have same secrets to import. let me know any suggestions or help.
    File "/usr/local/lib/python3.8/site-packages/prefect/client/secrets.py", line 137, in get
    value = secrets[self.name]
    ciaran

    ciaran

    1 year ago
    @Prabin Mehta Do you have an example of your flow? Or a cut down version that can re-produce this error?
    Prabin Mehta

    Prabin Mehta

    1 year ago
    @ciaran Below are my flows, this flow connect to snowflake and create a schema.1. This is the flow where I am passing the image and extra packages. This is running without an issue.
    with Flow(name="schema-generator",state_handlers=[handler],
              storage=GitHub(repo="prefect-flows",
                             path="flows/schema_generator.py",access_token_secret="GITHUB_ACCESS_TOKEN"),
              run_config=KubernetesRun(image="prefecthq/prefect:0.14.22-python3.8",
                                    env={"EXTRA_PIP_PACKAGES": "msgpack==1.0.0 lz4==3.1.1 numpy==1.18.1 prefect[snowflake]"}),
              executor=DaskExecutor("<tcp://dask-scheduler:8786>"),
              result=PrefectResult()) as flow_schema_generator:
        schema = Parameter("schema", default=[])
        _create_schema(schema=schema)
    2. This is the flow where I am passing ecr repo which have the prefect image installed with all dependencies. This is giving me the above mentioned error in job pods.
    with Flow(name="schema-generator",state_handlers=[handler],
              storage=GitHub(repo="prefect-flows",
                             path="flows/schema_generator.py",access_token_secret="GITHUB_ACCESS_TOKEN"),
              run_config=KubernetesRun(image="ecr_link/prefect-aura-image:latest",
                                       image_pull_secrets=["secret"]),
              executor=DaskExecutor("<tcp://dask-scheduler:8786>"),
              result=PrefectResult()) as flow_schema_generator:
        schema = Parameter("schema", default=[])
        _create_schema(schema=schema)
    ciaran

    ciaran

    1 year ago
    @Prabin Mehta have you seen this thread? https://prefect-community.slack.com/archives/CL09KU1K7/p1624284251299900?thread_ts=1624283518.297500&amp;cid=CL09KU1K7 @Tyler Wanner may have outlined what you need to do here.
    Prabin Mehta

    Prabin Mehta

    1 year ago
    Have used this to use ecr repo.
    ciaran

    ciaran

    1 year ago
    Did it work?
    Prabin Mehta

    Prabin Mehta

    1 year ago
    Thanks, It worked. Had to update the docker image with all dependencies and then while creating secret has to pass the full URL of the docker image to the docker-server parameter.
    ciaran

    ciaran

    1 year ago
    Glad it worked out!