Hi all, we have named volumes to which the agents are mounted to in prefect. We are facing an issue whenever a Flow Fails/Task fails involving a docker mount that the full server error message is returned with the full server values including themount location, user, address, and password. Obviously this is a massive security issue. Is there anyway to circumvent this output? SCRIPT ALERT: Intraday Strategy Greeks entered state Failed with message 500 Server Error for `http+docker://localhost/v1.41/containers/416e8db53c290dc44f8b1621c027e1930f07606101c82ed0ed059a367bcd8c13/start`: Internal Server Error ("error while mounting volume '/var/lib/docker/volumes/output/_data': failed to mount local volume: mount //XXXXXX.XXX.XXX/XXX:/XXX/lib/docker/volumes/output/_data, data: addr=XX.XXX.X.XX, username=XXXX, password=XXXXXX: no such file or directory"). (edited)

This is a docker-py log so we can’t control that, but I think what you can do to prevent sending this to us is attaching a Filter to your Python logger like this

The error isn't coming from Docker-py it is coming direclty from the docker.exe -> https://github.com/docker/cli/blob/21c5391cee93f90ca17fb841b60a8aa72373723c/opts/mount_test.go

Ah I see but I still think the best approach is the filter. Or does it fail before the Flow starts?

Yes as far as I know it is failing even before the flow begins

This is a bit tricky. I think you need to use script based storage and then attach a Python logging filter before the Flow so this log gets hidden. Or you could disable Prefect logs too.

I see that this allows writing files to docker storage but is there ability to read files from the docker storage as well?

Yeah. Do

build=False

when you register like this

Can we pass multiple storage params to the flow? Since our script is currently stored in CodeCommit but we want the outputs /inputs coming from mounting the volumes

Like if you want to use the Prefect interface to create an image, you can do:

docker = Docker(...)
docker.build()

and I think that will make the image. And then you just continue to use the CodeCommit storage on top of DockerRun. That’s what you mean right? But if you have the mounts also, the Flow should be able to just read them already

We already have the images created, we want to avoid the error message that is logged in prefect when starting a flow and the docker mount fails\

So to follow the filter suggestion on CodeCommit, you would attach the Python filter at the start of that Flow and hopefully it blocks that error log.

The error is not coming from Python it is a direct message from docker.exe. The python logger won't catch it

I agree, but if it appears in the Prefect logs, it must have gone through a Python logger so I think

docker-py

surfaced it.

Thanks for your patience through this. We will try a few more things.

Ohh ok that is a lot harder unfortunately. You may need to start the agent programatically instead of the CLI, and then before starting it, at a filter to the root logger? But then you don’t know why this fails either I think