I have a Flow I wish to register but that fails the Prefect healthcheck. It attempts to import a library that requires AWS credentials, which are then not found, and a Exception is raised, this causes the healthcheck to fail. I am using Docker Storage and and ECS Run Config. The reason it fails is that one of my Flow's dependencies uses AWS credentials that locally I provide via environment variables and that in production I provide by an EC2 instance metadata. I don't want to load the Flow, or the Docker image I use for Storage with any credential information for obvious reasons. How do I overcome this? I know about Prefect Secrets but it's something I want to avoid because it's unnecessary considering in production the credentials are available automatically using the EC2 metadata service. Health check fails at :

/opt/prefect/healthcheck.py

Hi @Hawkar Mahmod! Can you share a few pieces of info? It would be helpful to see: 1. the error you’re seeing 2 the specific docker image you’re using as the base image 3. Your Flow’s Run Config 4. Your Flow’s Storage configuration

Hey Dylan! Of course: 1. Error output

Step 9/9 : RUN python /opt/prefect/healthcheck.py '["/opt/prefect/flows/my-flow.prefect"]' '(3, 7)'
 ---> Running in 695588ba412f
Beginning health checks...
System Version check: OK
Traceback (most recent call last):
  File "/opt/prefect/healthcheck.py", line 151, in <module>
    flows = cloudpickle_deserialization_check(flow_file_paths)
  File "/opt/prefect/healthcheck.py", line 44, in cloudpickle_deserialization_check
    flows.append(cloudpickle.loads(flow_bytes))
  File "/usr/local/lib/python3.7/site-packages/cloudpickle/cloudpickle.py", line 562, in subimport
    __import__(name)
  File "/usr/local/lib/python3.7/site-packages/pingo/__init__.py", line 1, in <module>
    from pingo.api import *
  File "/usr/local/lib/python3.7/site-packages/pingo/api.py", line 1, in <module>
    from pingo.query import QueryRunner
  File "/usr/local/lib/python3.7/site-packages/pingo/query.py", line 14, in <module>
    db_credentials = json.loads(retrieve_parameter("some_param"))
  File "/usr/local/lib/python3.7/site-packages/pingo/utils.py", line 9, in retrieve_parameter
    WithDecryption=True
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 276, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 573, in _make_api_call
    operation_model, request_dict, request_context)
  File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 592, in _make_request
    return self._endpoint.make_request(operation_model, request_dict)
  File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 102, in make_request
    return self._send_request(request_dict, operation_model)
  File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 132, in _send_request
    request = self.create_request(request_dict, operation_model)
  File "/usr/local/lib/python3.7/site-packages/botocore/endpoint.py", line 116, in create_request
    operation_name=operation_model.name)
  File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 356, in emit
    return self._emitter.emit(aliased_event_name, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 228, in emit
    return self._emit(event_name, kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/hooks.py", line 211, in _emit
    response = handler(**kwargs)
  File "/usr/local/lib/python3.7/site-packages/botocore/signers.py", line 90, in handler
    return self.sign(operation_name, request)
  File "/usr/local/lib/python3.7/site-packages/botocore/signers.py", line 160, in sign
    auth.add_auth(request)
  File "/usr/local/lib/python3.7/site-packages/botocore/auth.py", line 357, in add_auth
    raise NoCredentialsError
botocore.exceptions.NoCredentialsError: Unable to locate credentials

1. I am using

prefecthq/prefect:0.14.6-python3.7

as the base image to create my own base image which is then added to by the Docker Storage. 3. Run Config:

flow.run_config = ECSRun(task_definition_path=str(PROJECT_ROOT / 'jobs/flow/task_definition.yml'))

4. Storage configuration:

flow.storage = Docker(registry_url=DOCKER_IMAGE_REGISTRY,
                      base_image=DOCKER_BASE_IMAGE,
                      image_name='prefect',
                      image_tag='img-001',
                      env_vars=dict(AWS_DEFAULT_REGION='eu-west-1'))

Hey @Hawkar Mahmod -- I don't think we have a straight-forward way to pull secrets from the AWS metadata service right now. You can pass environment variables to your ECS agent on startup with the

--env

flag https://docs.prefect.io/api/latest/cli/agent.html#ecs-start -- edit: may have missed your issue here, noticed you said it was working as intended with just the metadata!

Hey Michael, thanks for your response. That import is occurring inside of a library that I would prefer not to change. The metadata service is not a API service that AWS provides, it’s a runtime environment provided on EC2 instances when provisioned. It effectively provides AWS credentials to a whole EC2 instance and is not my concern in this discussion except to say that the Flow would work when I deploy it as Fargate. I just can’t reach that stage. I suppose what I was loooking to do to overcome thiis is to modify that healthcheck or switch it off, at least temporarily. Is that at all possible?

Hey Hawkar, you can always use a different base image for your Prefect Flow which doesn’t include the healthcheck

Ah yes! That may just have to be the solution. I know that in the docs it says all that’s required is that Prefect is installed. Other than that is there any concern with using a non-Prefect base image?

Not at all! Prefect base images have some things we find to be helpful most of the time but you can certainly use your own base image

Thanks @Dylan. For new users, the parameter to add is

ignore_healthchecks=True

to Flow(storage=Docker())