Hi, Any other users of MLFlow? I am trying load a model from S3 using MLFlow in a tasks, but it does not seem to be picking up the AWS credentials (passed through Prefect using PREFECT__CONTEXT__SECRETS__AWS_CREDENTIALS env var). Thanks!

Hi @Sébastien Arnaud! I’ve used MLFlow previously but not with Prefect. What error are you seeing?

If I run it locally on a local DaskExecutor no problem, but when I fire it on a remote executor(Coiled) it does not seem to be picking up the AWS_CREDENTIALS so I get a S3 Access Denied error:

File "/opt/conda/envs/coiled/lib/python3.9/site-packages/mlflow/store/artifact/s3_artifact_repo.py", line 110, in list_artifacts
    for result in results:
  File "/opt/conda/envs/coiled/lib/python3.9/site-packages/botocore/paginate.py", line 255, in __iter__
    response = self._make_request(current_kwargs)
  File "/opt/conda/envs/coiled/lib/python3.9/site-packages/botocore/paginate.py", line 332, in _make_request
    return self._method(**current_kwargs)
  File "/opt/conda/envs/coiled/lib/python3.9/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/opt/conda/envs/coiled/lib/python3.9/site-packages/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.exceptions.ClientError: An error occurred (AccessDenied) when calling the ListObjectsV2 operation: Access Denied

Gotcha and how are you passing it at the moment?

Simply setting up PREFECT__CONTEXT__SECRETS__AWS_CREDENTIALS since mlflow loads up boto3, I figured it would pass this along nicely

Are you using a LocalAgent to spin up the Coiled cluster?

I am spinning it up using this:

dask_executor = DaskExecutor(
            cluster_class=coiled.Cluster,
            cluster_kwargs={
                "configuration": "etl-cluster-config",
                "n_workers": 8,
            },
        )

Do you have a RunConfig?

Ok, cool - let me try this.

Assuming Coiled is a temporary cluster for your use case, you set the RunConfig for each flow to pass the env variable when the flow runs. The other way would be to do this on the Agent that is running the Flow.

prefect agent local start --env variable

. If you use it the PREFECT_CONTEXT_SECRETS way, I think you would need to configure these on the agent running the flow. I would pass it the way that MLFlow needs it which I’m not sure. Does that help?

Got it - thank you for the clarification!